You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2016/04/01 10:27:08 UTC

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 27 - Still Failing

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/27/

5 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=18654, name=Thread-8978, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=18654, name=Thread-8978, state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:57895/collection1
	at __randomizedtesting.SeedInfo.seed([509C900EC7537C02]:0)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:644)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:57895/collection1
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:642)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
	... 5 more


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([509C900EC7537C02:D8C8AFD469AF11FA]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:226)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
replicaCount expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: replicaCount expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([509C900EC7537C02:D8C8AFD469AF11FA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoConfigSetExist(CollectionsAPIDistributedZkTest.java:601)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 9 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 9 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([509C900EC7537C02]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:248)
	at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=5942, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=5942, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50456: collection already exists: awholynewstresscollection_collection0_1
	at __randomizedtesting.SeedInfo.seed([509C900EC7537C02]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:577)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1165)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:935)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:871)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1593)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1614)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)




Build Log:
[...truncated 11183 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/init-core-data-001
   [junit4]   2> 1    INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /is_s/
   [junit4]   2> 2795 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 5940 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 6905 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 6913 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 7104 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 7395 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_55331_hdfs____.7e1btx/webapp
   [junit4]   2> 7860 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 9044 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:55331
   [junit4]   2> 12400 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 12808 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 12829 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33566_datanode____at4hj0/webapp
   [junit4]   2> 13019 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 13739 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33566
   [junit4]   2> 15430 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 15433 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 15464 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47450_datanode____byk7kj/webapp
   [junit4]   2> 15815 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 17004 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[509C900EC7537C02]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47450
   [junit4]   2> 18348 INFO  (IPC Server handler 5 on 39532) [    ] BlockStateChange BLOCK* processReport: from storage DS-f8b4d8ce-2678-4169-9a1b-0086cce9b41f node DatanodeRegistration(127.0.0.1, datanodeUuid=09f82c23-cf66-4a37-a4d6-b4c7b4a81c03, infoPort=33566, ipcPort=33397, storageInfo=lv=-56;cid=testClusterID;nsid=947570425;c=0), blocks: 0, hasStaleStorages: true, processing time: 2 msecs
   [junit4]   2> 18348 INFO  (IPC Server handler 5 on 39532) [    ] BlockStateChange BLOCK* processReport: from storage DS-a4e05ab5-37d1-4f6b-b16e-fb982642bcbb node DatanodeRegistration(127.0.0.1, datanodeUuid=09f82c23-cf66-4a37-a4d6-b4c7b4a81c03, infoPort=33566, ipcPort=33397, storageInfo=lv=-56;cid=testClusterID;nsid=947570425;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 18351 INFO  (IPC Server handler 6 on 39532) [    ] BlockStateChange BLOCK* processReport: from storage DS-c328ed96-157b-4603-bb76-0b06a20d777c node DatanodeRegistration(127.0.0.1, datanodeUuid=5c5e4189-c87c-4a5b-bde1-87b0439b1e34, infoPort=47450, ipcPort=53454, storageInfo=lv=-56;cid=testClusterID;nsid=947570425;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 18355 INFO  (IPC Server handler 6 on 39532) [    ] BlockStateChange BLOCK* processReport: from storage DS-090bac70-9130-4c20-8787-e8b3a2112c31 node DatanodeRegistration(127.0.0.1, datanodeUuid=5c5e4189-c87c-4a5b-bde1-87b0439b1e34, infoPort=47450, ipcPort=53454, storageInfo=lv=-56;cid=testClusterID;nsid=947570425;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 20226 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 20227 INFO  (Thread-111) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 20228 INFO  (Thread-111) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 20331 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.ZkTestServer start zk server on port:36355
   [junit4]   2> 20380 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 20456 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 20502 INFO  (zkCallback-1-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4af3adc0 name:ZooKeeperConnection Watcher:127.0.0.1:36355 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20502 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 20506 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 20507 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 20533 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 20541 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 20550 INFO  (zkCallback-2-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@621cdace name:ZooKeeperConnection Watcher:127.0.0.1:36355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 20550 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 20551 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 20555 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 20564 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 20569 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 20573 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 20576 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 20587 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 20597 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 20597 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 20602 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 20602 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 20608 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 20608 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 20614 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 20614 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 20618 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 20618 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 20621 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 20621 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 20625 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 20625 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 20627 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 20627 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 20795 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 20796 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 20806 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 20807 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 20969 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores/collection1
   [junit4]   2> 20989 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.u.log Logging initialized @27012ms
   [junit4]   2> 21133 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 21213 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5f422113{/is_s,null,AVAILABLE}
   [junit4]   2> 21229 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.ServerConnector Started ServerConnector@427a92e8{HTTP/1.1,[http/1.1]}{127.0.0.1:41882}
   [junit4]   2> 21229 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.Server Started @27252ms
   [junit4]   2> 21230 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/is_s, hostPort=41882, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores}
   [junit4]   2> 21240 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 21257 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001'
   [junit4]   2> 21261 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 21261 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 21262 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 21264 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 21285 INFO  (zkCallback-3-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d6a390b name:ZooKeeperConnection Watcher:127.0.0.1:36355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21285 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 21286 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 21291 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x153d08c96a30002, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 21298 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 21300 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/solr.xml
   [junit4]   2> 21397 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores
   [junit4]   2> 21417 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.CoreContainer New CoreContainer 1018070149
   [junit4]   2> 21417 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001]
   [junit4]   2> 21418 WARN  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/lib
   [junit4]   2> 21445 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 21792 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 21795 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 21796 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 21797 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36355/solr
   [junit4]   2> 21809 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 21810 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 21815 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 21841 INFO  (zkCallback-6-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1e9f93d4 name:ZooKeeperConnection Watcher:127.0.0.1:36355 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21847 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 21847 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 21891 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 21900 INFO  (zkCallback-7-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3b274571 name:ZooKeeperConnection Watcher:127.0.0.1:36355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 21900 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 21906 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 21912 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 21917 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 21930 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 21934 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 21943 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 21945 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 21948 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 21949 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 21950 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 21971 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 21982 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 21984 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 22015 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 22022 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95649319346176004-127.0.0.1:41882_is_s-n_0000000000
   [junit4]   2> 22024 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41882_is_s
   [junit4]   2> 22024 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 22025 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer Overseer (id=95649319346176004-127.0.0.1:41882_is_s-n_0000000000) starting
   [junit4]   2> 22030 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 22061 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 22063 INFO  (OverseerCollectionConfigSetProcessor-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 22068 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41882_is_s
   [junit4]   2> 22071 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:41882_is_s
   [junit4]   2> 22079 INFO  (zkCallback-7-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 22084 INFO  (zkCallback-7-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 22238 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 22238 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 22274 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores
   [junit4]   2> 22281 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 22281 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores/collection1
   [junit4]   2> 22281 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 22296 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 22296 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [n:127.0.0.1:41882_is_s    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 22299 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 22299 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 22304 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 22339 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 22365 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 22368 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41882/is_s",
   [junit4]   2>   "node_name":"127.0.0.1:41882_is_s",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 22383 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41882/is_s",
   [junit4]   2>   "node_name":"127.0.0.1:41882_is_s",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 22396 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 22410 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 22419 INFO  (zkCallback-7-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 23305 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 23305 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 23306 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 23306 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 23308 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 23309 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores/collection1'
   [junit4]   2> 23312 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 23312 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 23317 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 23370 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 23416 WARN  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 23429 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 23750 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 24158 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 25530 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 25546 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 25548 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 25986 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 25989 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 25991 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 25993 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 25996 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 25997 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 26029 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 26038 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:39532/solr_hdfs_home
   [junit4]   2> 26038 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 26047 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/control-001/cores/collection1], dataDir=[hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 26063 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3212e247
   [junit4]   2> 26112 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 26490 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 26508 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 26731 WARN  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:39532/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 26812 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 26861 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 26861 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 26861 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 30093 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 30102 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 30105 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@37755308
   [junit4]   2> 30955 INFO  (IPC Server handler 4 on 39532) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:60265 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-090bac70-9130-4c20-8787-e8b3a2112c31:NORMAL:127.0.0.1:44610|RBW], ReplicaUnderConstruction[[DISK]DS-f8b4d8ce-2678-4169-9a1b-0086cce9b41f:NORMAL:127.0.0.1:60265|RBW]]} size 71
   [junit4]   2> 30956 INFO  (IPC Server handler 6 on 39532) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:44610 is added to blk_1073741825_1001 size 71
   [junit4]   2> 31415 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@709dc75c); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 31417 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 31462 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 32078 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 32113 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 32128 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 32129 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 32129 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 32129 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 32393 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 32394 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 32394 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 32400 WARN  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 32431 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 32441 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 32442 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 32443 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 32664 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 32665 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 32688 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 32688 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 32688 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 32725 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 32725 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 32732 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=4, maxMergeAtOnceExplicit=7, maxMergedSegmentMB=62.291015625, floorSegmentMB=0.744140625, forceMergeDeletesPctAllowed=14.524766021452072, segmentsPerTier=40.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7651711111182025
   [junit4]   2> 32876 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:39532/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@709dc75c); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 32876 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 32910 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@633942cd[collection1] main]
   [junit4]   2> 32943 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 32957 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 32957 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 32959 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 32959 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 32995 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 32996 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 32996 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 32996 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 33049 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 33136 INFO  (searcherExecutor-9-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@633942cd[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 33146 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 33147 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 33147 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 33148 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1530389123037134848
   [junit4]   2> 33161 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 13.0ms to seed version buckets with highest version 1530389123037134848
   [junit4]   2> 33276 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 33279 INFO  (coreLoadExecutor-8-thread-1-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 33283 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:41882/is_s collection:control_collection shard:shard1
   [junit4]   2> 33292 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 33325 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95649319346176004-core_node1-n_0000000000
   [junit4]   2> 33325 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 33327 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 33327 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 33328 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41882/is_s/collection1/
   [junit4]   2> 33328 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 33328 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:41882/is_s/collection1/ has no replicas
   [junit4]   2> 33329 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 33336 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 33348 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95649319346176004-core_node1-n_0000000000
   [junit4]   2> 33355 INFO  (zkCallback-7-thread-2-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 33366 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:41882/is_s",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 33373 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41882/is_s/collection1/ shard1
   [junit4]   2> 33477 INFO  (zkCallback-7-thread-2-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 33526 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:41882/is_s/collection1/ and leader is http://127.0.0.1:41882/is_s/collection1/
   [junit4]   2> 33527 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:41882/is_s
   [junit4]   2> 33527 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 33527 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 33527 INFO  (coreZkRegister-2-thread-1-processing-n:127.0.0.1:41882_is_s x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:41882_is_s c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 33530 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41882/is_s",
   [junit4]   2>   "node_name":"127.0.0.1:41882_is_s",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 33531 INFO  (OverseerStateUpdate-95649319346176004-127.0.0.1:41882_is_s-n_0000000000) [n:127.0.0.1:41882_is_s    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:41882/is_s",
   [junit4]   2>   "node_name":"127.0.0.1:41882_is_s",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 33565 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 33579 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 33607 INFO  (zkCallback-9-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5ae28b2e name:ZooKeeperConnection Watcher:127.0.0.1:36355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 33607 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 33608 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 33608 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 33812 INFO  (zkCallback-9-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 33812 INFO  (zkCallback-7-thread-2-processing-n:127.0.0.1:41882_is_s) [n:127.0.0.1:41882_is_s    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 33822 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 33833 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 33956 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/shard-1-001/cores/collection1
   [junit4]   2> 33957 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/shard-1-001
   [junit4]   2> 33960 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 34003 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@600afa4c{/is_s,null,AVAILABLE}
   [junit4]   2> 34004 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.ServerConnector Started ServerConnector@33c4f01d{HTTP/1.1,[http/1.1]}{127.0.0.1:54147}
   [junit4]   2> 34004 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.e.j.s.Server Started @40027ms
   [junit4]   2> 34004 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solrconfig=solrconfig.xml, hostContext=/is_s, hostPort=54147, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/shard-1-001/cores}
   [junit4]   2> 34004 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 34004 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_509C900EC7537C02-001/shard-1-001'
   [junit4]   2> 34005 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 34005 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 34005 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 34042 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 34099 INFO  (zkCallback-10-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3273a50b name:ZooKeeperConnection Watcher:127.0.0.1:36355/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 34099 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 34100 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 34130 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 34130 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[509C900EC7537C02]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.

[...truncated too long message...]

CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:351)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:386)
   [junit4]   2> 	at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:270)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:387)
   [junit4]   2> 	at org.apache.solr.handler.ReplicationHandler$1.run(ReplicationHandler.java:275)
   [junit4]   2> 
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=de-AT, timezone=Canada/Saskatchewan
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=273077912,total=521142272
   [junit4]   2> NOTE: All tests run in this JVM: [SharedFSAutoReplicaFailoverTest, AssignTest, SuggesterTSTTest, TestComplexPhraseQParserPlugin, AnalyticsMergeStrategyTest, CollectionsAPISolrJTest, SpatialHeatmapFacetsTest, DirectSolrConnectionTest, NoCacheHeaderTest, DirectUpdateHandlerOptimizeTest, TestCollectionAPI, FieldMutatingUpdateProcessorTest, TestReversedWildcardFilterFactory, TestFieldTypeCollectionResource, FieldAnalysisRequestHandlerTest, OverseerTest, TestClusterStateMutator, TestNamedUpdateProcessors, TlogReplayBufferedWhileIndexingTest, TestJmxMonitoredMap, SuggesterWFSTTest, TestDefaultStatsCache, DistribJoinFromCollectionTest, TestDynamicLoading, SliceStateTest, TestJmxIntegration, PluginInfoTest, TestFuzzyAnalyzedSuggestions, HdfsChaosMonkeySafeLeaderTest, TestOrdValues, TestXIncludeConfig, TestStressReorder, TestTrackingShardHandlerFactory, TestFastWriter, TestFileDictionaryLookup, QueryParsingTest, LeaderElectionTest, VersionInfoTest, TestConfigReload, BadIndexSchemaTest, UpdateParamsTest, TestConfigSetsAPIZkFailure, TestSolrJ, TestSolrIndexConfig, TestDownShardTolerantSearch, TestPerFieldSimilarityClassic, SmileWriterTest, BlockCacheTest, BadComponentTest, AnalysisErrorHandlingTest, TestAddFieldRealTimeGet, RegexBoostProcessorTest, TestExactStatsCache, HdfsUnloadDistributedZkTest, DebugComponentTest, TestReload, HdfsDirectoryTest, DocValuesMultiTest, UnloadDistributedZkTest, TestCSVResponseWriter, TestJettySolrRunner, SimpleFacetsTest, BadCopyFieldTest, HdfsSyncSliceTest, TestIndexSearcher, TestSchemaResource, IgnoreCommitOptimizeUpdateProcessorFactoryTest, CollectionStateFormat2Test, TestSolrXml, HardAutoCommitTest, MultiThreadedOCPTest, TestCoreDiscovery, TestClassicSimilarityFactory, TestSuggestSpellingConverter, TestChildDocTransformer, TestRebalanceLeaders, SearchHandlerTest, ReplicationFactorTest, ClusterStateTest, StatelessScriptUpdateProcessorFactoryTest, BigEndianAscendingWordSerializerTest, WrapperMergePolicyFactoryTest, TestCloudSchemaless, SSLMigrationTest, TestReloadDeadlock, ScriptEngineTest, OverseerCollectionConfigSetProcessorTest, ShowFileRequestHandlerTest, TestElisionMultitermQuery, PKIAuthenticationIntegrationTest, TestRequestStatusCollectionAPI, TestStandardQParsers, TestCollationField, CachingDirectoryFactoryTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, OpenCloseCoreStressTest, TestReplicationHandler]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestReplicationHandler -Dtests.seed=509C900EC7537C02 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=de-AT -Dtests.timezone=Canada/Saskatchewan -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestReplicationHandler (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 9 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([509C900EC7537C02]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:248)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [441/585 (5!)] on J2 in 1070.73s, 13 tests, 1 failure <<< FAILURES!

[...truncated 491 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2360 tests, 1 suite-level error, 2 errors, 2 failures, 31 ignored (16 assumptions) [seed: 509C900EC7537C02]

Total time: 315 minutes 17 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 58 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/58/

7 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
document count mismatch.  control=3941 sum(shards)=3940 cloudClient=3940

Stack Trace:
java.lang.AssertionError: document count mismatch.  control=3941 sum(shards)=3940 cloudClient=3940
	at __randomizedtesting.SeedInfo.seed([8F44261516A7A083:71019CFB85BCD7B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1319)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:228)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.TermVectorComponentDistributedTest

Error Message:


Stack Trace:
java.lang.AssertionError
	at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.before(TestRuleTemporaryFilesCleanup.java:112)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.before(TestRuleAdapter.java:26)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:35)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.component.TermVectorComponentDistributedTest

Error Message:
71 threads leaked from SUITE scope at org.apache.solr.handler.component.TermVectorComponentDistributedTest:     1) Thread[id=8465, name=coreZkRegister-1049-thread-18-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    2) Thread[id=8523, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data3/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    3) Thread[id=8524, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data1/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    4) Thread[id=8451, name=coreZkRegister-1049-thread-4-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    5) Thread[id=8468, name=coreZkRegister-1049-thread-21-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    6) Thread[id=8498, name=coreZkRegister-1208-thread-19, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    7) Thread[id=8512, name=updateExecutor-493-thread-6-processing-n:127.0.0.1:50321_ x:awholynewstresscollection_collection4_0_shard1_replica4 s:shard1 c:awholynewstresscollection_collection4_0 r:core_node6, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)         at java.util.concurrent.FutureTask.get(FutureTask.java:191)         at org.apache.solr.update.DefaultSolrCoreState$2.run(DefaultSolrCoreState.java:317)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    8) Thread[id=8172, name=java.util.concurrent.ThreadPoolExecutor$Worker@3c2734e6[State = -1, empty queue], state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)    9) Thread[id=8475, name=OverseerThreadFactory-1932-thread-4, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   10) Thread[id=8489, name=coreZkRegister-1208-thread-10, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   11) Thread[id=8457, name=coreZkRegister-1049-thread-10, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   12) Thread[id=8494, name=coreZkRegister-1208-thread-15-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1000)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:970)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:926)         at org.apache.solr.cloud.ZkController.register(ZkController.java:865)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   13) Thread[id=8438, name=qtp1292668453-8438, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)   14) Thread[id=8493, name=coreZkRegister-1208-thread-14-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1000)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:970)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:926)         at org.apache.solr.cloud.ZkController.register(ZkController.java:865)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   15) Thread[id=8491, name=coreZkRegister-1208-thread-12-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1000)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:970)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:926)         at org.apache.solr.cloud.ZkController.register(ZkController.java:865)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   16) Thread[id=8462, name=coreZkRegister-1049-thread-15-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   17) Thread[id=8418, name=IPC Parameter Sending Thread #38, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   18) Thread[id=8499, name=coreZkRegister-1208-thread-20, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   19) Thread[id=8520, name=zkCallback-460-thread-17, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   20) Thread[id=8455, name=coreZkRegister-1049-thread-8, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   21) Thread[id=8488, name=coreZkRegister-1208-thread-9-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   22) Thread[id=8477, name=OverseerThreadFactory-1932-thread-6, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   23) Thread[id=8444, name=OverseerStateUpdate-95850796072239334-127.0.0.1:50321_-n_0000000006, state=TIMED_WAITING, group=Overseer state updater.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:168)         at org.apache.solr.cloud.DistributedQueue.peek(DistributedQueue.java:142)         at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:196)         at java.lang.Thread.run(Thread.java:745)   24) Thread[id=8452, name=coreZkRegister-1049-thread-5-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   25) Thread[id=8486, name=coreZkRegister-1208-thread-7-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   26) Thread[id=8443, name=qtp1208954681-8443, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)   27) Thread[id=8449, name=zkCallback-465-thread-12, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   28) Thread[id=8476, name=OverseerThreadFactory-1932-thread-5, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   29) Thread[id=8514, name=recoveryExecutor-494-thread-6-processing-n:127.0.0.1:50321_ x:awholynewstresscollection_collection4_0_shard1_replica4 s:shard1 c:awholynewstresscollection_collection4_0 r:core_node6, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:675)         at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:663)         at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:305)         at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:221)         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   30) Thread[id=8484, name=coreZkRegister-1208-thread-5-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1000)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:970)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:926)         at org.apache.solr.cloud.ZkController.register(ZkController.java:865)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   31) Thread[id=8474, name=OverseerThreadFactory-1932-thread-3, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   32) Thread[id=8471, name=coreZkRegister-1049-thread-23, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   33) Thread[id=8415, name=IPC Client (2081512138) connection to localhost/127.0.0.1:53213 from jenkins, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Object.wait(Native Method)         at org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:920)         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:965)   34) Thread[id=8473, name=OverseerThreadFactory-1932-thread-2, state=WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   35) Thread[id=8464, name=coreZkRegister-1049-thread-17-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   36) Thread[id=8496, name=coreZkRegister-1208-thread-17-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   37) Thread[id=8490, name=coreZkRegister-1208-thread-11, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   38) Thread[id=8417, name=IPC Parameter Sending Thread #37, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   39) Thread[id=8485, name=coreZkRegister-1208-thread-6, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   40) Thread[id=8469, name=coreZkRegister-1049-thread-22-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   41) Thread[id=8441, name=qtp460475988-8441, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:745)   42) Thread[id=8447, name=zkCallback-496-thread-20, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   43) Thread[id=8448, name=coreZkRegister-1049-thread-2-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   44) Thread[id=8459, name=coreZkRegister-1049-thread-12, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   45) Thread[id=8497, name=coreZkRegister-1208-thread-18-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   46) Thread[id=8461, name=coreZkRegister-1049-thread-14-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   47) Thread[id=8467, name=coreZkRegister-1049-thread-20-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   48) Thread[id=8435, name=qtp199004788-8435, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)   49) Thread[id=8445, name=OverseerCollectionConfigSetProcessor-95850796072239334-127.0.0.1:50321_-n_0000000006, state=TIMED_WAITING, group=Overseer collection creation process.]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.apache.solr.cloud.DistributedQueue.getChildren(DistributedQueue.java:342)         at org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:235)         at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:200)         at java.lang.Thread.run(Thread.java:745)   50) Thread[id=8442, name=qtp1208954681-8442, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:233)         at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:181)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:171)         at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)         at java.lang.Thread.run(Thread.java:745)   51) Thread[id=8437, name=qtp1292668453-8437, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)         at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:546)         at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:47)         at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:609)         at java.lang.Thread.run(Thread.java:745)   52) Thread[id=8522, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data4/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   53) Thread[id=8463, name=coreZkRegister-1049-thread-16-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   54) Thread[id=8466, name=coreZkRegister-1049-thread-19, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   55) Thread[id=8521, name=zkCallback-455-thread-15, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   56) Thread[id=8458, name=coreZkRegister-1049-thread-11, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   57) Thread[id=8453, name=coreZkRegister-1049-thread-6-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   58) Thread[id=8454, name=coreZkRegister-1049-thread-7, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   59) Thread[id=8487, name=coreZkRegister-1208-thread-8-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)         at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)         at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)         at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)         at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)         at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)         at org.apache.solr.cloud.ZkController.register(ZkController.java:853)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   60) Thread[id=8450, name=coreZkRegister-1049-thread-3, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   61) Thread[id=8495, name=coreZkRegister-1208-thread-16-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1000)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:970)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:926)         at org.apache.solr.cloud.ZkController.register(ZkController.java:865)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   62) Thread[id=8492, name=coreZkRegister-1208-thread-13-processing-n:127.0.0.1:50321_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:1000)         at org.apache.solr.cloud.ZkController.getLeaderProps(ZkController.java:970)         at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:926)         at org.apache.solr.cloud.ZkController.register(ZkController.java:865)         at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)         at java.util.concurrent.FutureTask.run(FutureTask.java:266)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)         at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   63) Thread[id=8158, name=IPC Client (2081512138) connection to localhost/127.0.0.1:53213 from jenkins, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)         at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)         at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)         at org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335)         at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157)         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)         at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)         at java.io.FilterInputStream.read(FilterInputStream.java:133)         at java.io.FilterInputStream.read(FilterInputStream.java:133)         at org.apache.hadoop.ipc.Client$Connection$PingInputStream.read(Client.java:513)         at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)         at java.io.BufferedInputStream.read(BufferedInputStream.java:265)         at java.io.DataInputStream.readInt(DataInputStream.java:387)         at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1071)         at org.apache.hadoop.ipc.Client$Connection.run(Client.java:966)   64) Thread[id=8470, name=zkCallback-465-thread-13, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   65) Thread[id=8483, name=coreZkRegister-1208-thread-4, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   66) Thread[id=8446, name=OverseerHdfsCoreFailoverThread-95850796072239334-127.0.0.1:50321_-n_0000000006, state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]         at java.lang.Thread.sleep(Native Method)         at org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:138)         at java.lang.Thread.run(Thread.java:745)   67) Thread[id=8516, name=updateExecutor-493-thread-7, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   68) Thread[id=8460, name=coreZkRegister-1049-thread-13, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   69) Thread[id=8456, name=coreZkRegister-1049-thread-9, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   70) Thread[id=8518, name=recoveryExecutor-494-thread-7, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)   71) Thread[id=8525, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data2/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 71 threads leaked from SUITE scope at org.apache.solr.handler.component.TermVectorComponentDistributedTest: 
   1) Thread[id=8465, name=coreZkRegister-1049-thread-18-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
        at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
        at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:853)
        at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=8523, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data3/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   3) Thread[id=8524, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data1/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   4) Thread[id=8451, name=coreZkRegister-1049-thread-4-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
        at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
        at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:853)
        at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   5) Thread[id=8468, name=coreZkRegister-1049-thread-21-processing-n:127.0.0.1:37006_, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.solr.cloud.ShardLeaderElectionContext.waitForReplicasToComeUp(ElectionContext.java:616)
        at org.apache.solr.cloud.ShardLeaderElectionContext.runLeaderProcess(ElectionContext.java:318)
        at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:170)
        at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:135)
        at org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:307)
        at org.apache.solr.cloud.ZkController.joinElection(ZkController.java:1042)
        at org.apache.solr.cloud.ZkController.register(ZkController.java:853)
        at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:203)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   6) Thread[id=8498, name=coreZkRegister-1208-thread-19, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   7) Thread[id=8512, name=updateExecutor-493-thread-6-processing-n:127.0.0.1:50321_ x:awholynewstresscollection_collection4_0_shard1_replica4 s:shard1 c:awholynewstresscollection_collection4_0 r:core_node6, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:429)
        at java.util.concurrent.FutureTask.get(FutureTask.java:191)
        at org.apache.solr.update.DefaultSolrCoreState$2.run(DefaultSolrCoreState.java:317)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
        at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1049045488.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   8) Thread[id=8172, name=java.util.concurrent.ThreadPoolExecutor$Worker@3c2734e6[State = -1, empty queue], state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
   9) Threa

[...truncated too long message...]

_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   52) Thread[id=8499, name=coreZkRegister-1208-thread-20, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   53) Thread[id=8458, name=coreZkRegister-1049-thread-11, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   54) Thread[id=8453, name=coreZkRegister-1049-thread-6, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   55) Thread[id=8464, name=coreZkRegister-1049-thread-17, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   56) Thread[id=8518, name=recoveryExecutor-494-thread-7, state=TIMED_WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
   [junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
   [junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    >   57) Thread[id=8525, name=Async disk worker #2 for volume /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_8F44261516A7A083-001/tempDir-001/hdfsBaseDir/data/data2/current, state=TIMED_WAITING, group=FsDatasetAsyncDiskService]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([8F44261516A7A083]:0)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2833, name=Thread-1066, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap spaceThrowable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=2753, name=IPC Server idle connection scanner for port 53213, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4] Completed [519/595 (4!)] on J2 in 8163.44s, 0 tests, 1 failure, 4 errors <<< FAILURES!

[...truncated 229 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J2-20160506_203053_666.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid10577.hprof ...
   [junit4] Heap dump file created [682066696 bytes in 5.571 secs]
   [junit4] <<< JVM J2: EOF ----

[...truncated 13 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 595 suites (10 ignored), 2403 tests, 6 suite-level errors, 1 error, 1 failure, 361 ignored (13 assumptions) [seed: 8F44261516A7A083]

Total time: 558 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 57 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/57/

4 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:58336

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:58336
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:457)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest:     1) Thread[id=7672, name=searcherExecutor-2383-thread-1, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest: 
   1) Thread[id=7672, name=searcherExecutor-2383-thread-1, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E363401D1F6FC7BD]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=7672, name=searcherExecutor-2383-thread-1, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=7672, name=searcherExecutor-2383-thread-1, state=WAITING, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([E363401D1F6FC7BD]:0)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=115839, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=115839, name=collection0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:59048: Could not find collection : awholynewstresscollection_collection0_2
	at __randomizedtesting.SeedInfo.seed([E363401D1F6FC7BD]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:994)




Build Log:
[...truncated 11450 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/init-core-data-001
   [junit4]   2> 1250651 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[E363401D1F6FC7BD]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1250651 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[E363401D1F6FC7BD]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1250656 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1250678 INFO  (Thread-3866) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1250678 INFO  (Thread-3866) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1250770 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.ZkTestServer start zk server on port:54018
   [junit4]   2> 1250770 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250794 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250802 INFO  (zkCallback-30613-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@228fb82b name:ZooKeeperConnection Watcher:127.0.0.1:54018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250802 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250802 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250802 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1250814 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250826 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250842 INFO  (zkCallback-30614-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d376033 name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250842 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250843 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250843 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1250843 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1250844 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1250844 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1250845 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1250845 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1250845 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1250846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1250846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1250846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1250847 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1250847 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1250847 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1250848 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1250848 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1250848 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1250849 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1250849 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1250849 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1250849 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1250850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1250850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1250850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1250850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1250851 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1250851 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1250852 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250866 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250879 INFO  (zkCallback-30615-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@29181fe7 name:ZooKeeperConnection Watcher:127.0.0.1:54018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250879 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250879 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250879 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1250880 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250880 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250881 INFO  (zkCallback-30616-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4db05ded name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250881 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250881 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250881 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1250881 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1250882 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1250883 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 1250883 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 1250883 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 1250884 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1250884 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1250885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1250885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1250886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1250886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1250886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1250887 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 1250887 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1250887 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1250888 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1250888 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1250889 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1250889 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1250889 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1250890 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1250890 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1250890 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1250947 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1
   [junit4]   2> 1250948 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1250949 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1fbb6727{/,null,AVAILABLE}
   [junit4]   2> 1250974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.ServerConnector Started ServerConnector@34c8d22{HTTP/1.1,[http/1.1]}{127.0.0.1:33749}
   [junit4]   2> 1250974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.Server Started @1253105ms
   [junit4]   2> 1250974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/tempDir-001/control/data, hostContext=/, hostPort=33749, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores}
   [junit4]   2> 1250975 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1250975 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001'
   [junit4]   2> 1250975 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1250975 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1250975 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250980 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250990 INFO  (zkCallback-30617-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@269a429d name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250990 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1250991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/solr.xml
   [junit4]   2> 1250995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores
   [junit4]   2> 1250995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.CoreContainer New CoreContainer 838482464
   [junit4]   2> 1250995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001]
   [junit4]   2> 1250995 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/lib
   [junit4]   2> 1250995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1250996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1250996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1250996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1250996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54018/solr
   [junit4]   2> 1250996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1250996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1251006 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1251012 INFO  (zkCallback-30620-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2c8961e2 name:ZooKeeperConnection Watcher:127.0.0.1:54018 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1251012 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1251013 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1251014 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/1553166551@2eaeef44
   [junit4]   2> 1251017 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1251023 INFO  (zkCallback-30621-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@661db907 name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1251023 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1251024 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1251024 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1251025 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1251026 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1251026 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1251027 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1251027 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1251028 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1251028 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1251028 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1251029 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1251029 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1251029 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1251030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1251030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95844247090429958-127.0.0.1:33749_-n_0000000000
   [junit4]   2> 1251030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33749_
   [junit4]   2> 1251030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1251030 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer Overseer (id=95844247090429958-127.0.0.1:33749_-n_0000000000) starting
   [junit4]   2> 1251031 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1251033 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1251034 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1251036 INFO  (OverseerCollectionConfigSetProcessor-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1251039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33749_
   [junit4]   2> 1251039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:33749_
   [junit4]   2> 1251040 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1251040 INFO  (zkCallback-30621-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1251040 INFO  (zkCallback-30621-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1251047 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1251047 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1251048 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores
   [junit4]   2> 1251048 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1251048 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1
   [junit4]   2> 1251048 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1251051 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1251051 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1251051 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1251051 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 1251052 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 1251052 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1251053 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33749",
   [junit4]   2>   "node_name":"127.0.0.1:33749_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1251053 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33749",
   [junit4]   2>   "node_name":"127.0.0.1:33749_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1251053 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1251053 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1251054 INFO  (zkCallback-30621-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1251054 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1251054 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [n:127.0.0.1:33749_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1252052 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1252052 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1252052 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1252053 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1252053 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1252053 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1'
   [junit4]   2> 1252053 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1252053 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1252054 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1252057 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1252060 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1252086 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1252090 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1252146 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1252150 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1252151 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1252155 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1252156 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1252157 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1252157 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1252158 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1252158 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1252158 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1252158 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data/]
   [junit4]   2> 1252159 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6815ed1d
   [junit4]   2> 1252159 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data
   [junit4]   2> 1252159 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1252159 WARN  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1252159 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data/index
   [junit4]   2> 1252160 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=41, maxMergeAtOnceExplicit=41, maxMergedSegmentMB=23.4150390625, floorSegmentMB=1.994140625, forceMergeDeletesPctAllowed=4.78710904714002, segmentsPerTier=44.0, maxCFSSegmentSizeMB=1.3779296875, noCFSRatio=0.361575544003812
   [junit4]   2> 1252183 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56497789),segFN=segments_1,generation=1}
   [junit4]   2> 1252183 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1252193 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1252193 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1252194 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1252194 WARN  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1252195 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1252196 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1252196 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1252197 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1252198 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1252199 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1252202 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data/
   [junit4]   2> 1252202 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1252212 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1252212 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1252213 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1252213 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1252213 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@20f9c9aa
   [junit4]   2> 1252214 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(SimpleFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/index-SimpleFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@56497789),segFN=segments_1,generation=1}
   [junit4]   2> 1252214 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1252215 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3d00526b[collection1] main]
   [junit4]   2> 1252215 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1252215 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1252215 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1252216 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1252217 INFO  (searcherExecutor-7106-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3d00526b[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1252217 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1252217 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1252217 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1252217 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1533507955049103360
   [junit4]   2> 1252218 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1533507955049103360
   [junit4]   2> 1252218 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1252219 INFO  (coreLoadExecutor-7105-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1252224 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:33749 collection:control_collection shard:shard1
   [junit4]   2> 1252224 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1252235 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95844247090429958-core_node1-n_0000000000
   [junit4]   2> 1252236 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1252236 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1252237 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1252237 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1252237 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33749/collection1/
   [junit4]   2> 1252237 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1252237 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:33749/collection1/ has no replicas
   [junit4]   2> 1252239 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1252239 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95844247090429958-core_node1-n_0000000000
   [junit4]   2> 1252240 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33749/collection1/ shard1
   [junit4]   2> 1252240 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:33749",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1252459 INFO  (zkCallback-30621-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252508 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:33749/collection1/ and leader is http://127.0.0.1:33749/collection1/
   [junit4]   2> 1252508 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33749
   [junit4]   2> 1252508 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1252508 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1252508 INFO  (coreZkRegister-7099-thread-1-processing-n:127.0.0.1:33749_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33749_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1252509 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33749",
   [junit4]   2>   "node_name":"127.0.0.1:33749_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1252514 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33749",
   [junit4]   2>   "node_name":"127.0.0.1:33749_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1252555 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:54018/solr
   [junit4]   2> 1252556 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1252582 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1252609 INFO  (zkCallback-30623-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6b532077 name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1252609 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1252609 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1252609 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1252609 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1252610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1252610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1252610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1252615 INFO  (zkCallback-30621-thread-1-processing-n:127.0.0.1:33749_) [n:127.0.0.1:33749_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252630 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1252630 INFO  (zkCallback-30623-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252643 INFO  (zkCallback-30624-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@39379216 name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1252644 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1252644 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1252645 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 1252645 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1252747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/shard-1-001/cores/collection1
   [junit4]   2> 1252747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/shard-1-001
   [junit4]   2> 1252748 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1252752 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7bc53241{/,null,AVAILABLE}
   [junit4]   2> 1252752 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.ServerConnector Started ServerConnector@231ad273{HTTP/1.1,[http/1.1]}{127.0.0.1:44656}
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.e.j.s.Server Started @1254884ms
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=44656, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/shard-1-001/cores}
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/shard-1-001'
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1252753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1252754 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1252754 INFO  (zkCallback-30625-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3c8b7886 name:ZooKeeperConnection Watcher:127.0.0.1:54018/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1252754 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1252755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1252755 INFO  (OverseerStateUpdate-95844247090429958-127.0.0.1:33749_-n_0000000000) [n:127.0.0.1:33749_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 1252755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1252755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/shard-1-001/solr.xml
   [junit4]   2> 1252767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_E363401D1F6FC7BD-001/shard-1-001/cores
   [junit4]   2> 1252767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E363401D1F6FC7BD]) [    ] o.a.s.c.CoreContainer New CoreContainer 1387781136
   [junit4]   2> 1252767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[E3634

[...truncated too long message...]

-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/icu/lucene-analyzers-icu-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J1-20160505_163136_347.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J1-20160505_163136_347.suites -stdin
   [junit4] ERROR: JVM J1 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 245 minutes 24 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 56 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/56/

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=17253, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=17253, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39126: collection already exists: awholynewstresscollection_collection1_0
	at __randomizedtesting.SeedInfo.seed([358C62A1A273FF2D]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:987)




Build Log:
[...truncated 11942 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/init-core-data-001
   [junit4]   2> 2210492 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[358C62A1A273FF2D]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 2210492 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[358C62A1A273FF2D]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2210494 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2210494 INFO  (Thread-6400) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2210494 INFO  (Thread-6400) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2210594 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ZkTestServer start zk server on port:57646
   [junit4]   2> 2210594 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2210594 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210596 INFO  (zkCallback-2424-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d7a83b4 name:ZooKeeperConnection Watcher:127.0.0.1:57646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210596 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210596 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2210596 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2210597 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2210598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210598 INFO  (zkCallback-2425-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46c47bd5 name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2210598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 2210600 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 2210600 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 2210601 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 2210602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2210602 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2210605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2210605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 2210606 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2210606 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2210609 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2210609 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2210610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2210610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2210611 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2210611 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 2210612 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2210612 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2210613 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2210613 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2210615 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2210615 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2210616 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2210616 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2210617 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2210617 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2210620 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2210621 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210622 INFO  (zkCallback-2426-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4e49abd5 name:ZooKeeperConnection Watcher:127.0.0.1:57646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210622 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210623 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2210623 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2210624 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2210624 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210625 INFO  (zkCallback-2427-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5adfb46a name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210625 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210625 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2210625 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 2210625 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 2210627 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 2210627 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 2210628 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 2210628 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 2210630 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2210630 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2210631 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 2210631 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 2210634 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 2210634 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 2210635 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 2210635 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 2210638 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 2210638 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 2210640 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2210640 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2210641 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 2210641 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 2210642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 2210642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 2210644 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 2210644 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 2210704 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1
   [junit4]   2> 2210705 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 2210706 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2c116764{/,null,AVAILABLE}
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6ecdc687{HTTP/1.1,[http/1.1]}{127.0.0.1:59752}
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.Server Started @2212974ms
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/tempDir-001/control/data, hostContext=/, hostPort=59752, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores}
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001'
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2210708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2210709 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210709 INFO  (zkCallback-2428-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1a550be5 name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210709 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2210710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2210710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/solr.xml
   [junit4]   2> 2210714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores
   [junit4]   2> 2210714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CoreContainer New CoreContainer 1685680505
   [junit4]   2> 2210714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001]
   [junit4]   2> 2210714 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/lib
   [junit4]   2> 2210714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2210715 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2210715 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2210716 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2210716 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57646/solr
   [junit4]   2> 2210716 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2210716 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2210716 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210719 INFO  (zkCallback-2431-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@10890172 name:ZooKeeperConnection Watcher:127.0.0.1:57646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210719 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210719 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2210720 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$27/1828077232@4777a1d9
   [junit4]   2> 2210721 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2210722 INFO  (zkCallback-2432-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5333b36b name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2210722 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2210723 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2210724 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 2210726 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 2210727 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 2210728 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 2210731 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 2210732 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 2210733 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 2210735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 2210736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2210737 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 2210739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2210740 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 2210741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2210741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95839401293905926-127.0.0.1:59752_-n_0000000000
   [junit4]   2> 2210741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:59752_
   [junit4]   2> 2210741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 2210742 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer Overseer (id=95839401293905926-127.0.0.1:59752_-n_0000000000) starting
   [junit4]   2> 2210743 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2210747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2210747 INFO  (OverseerCollectionConfigSetProcessor-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 2210747 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:59752_
   [junit4]   2> 2210748 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:59752_
   [junit4]   2> 2210748 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2210750 INFO  (zkCallback-2432-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2210757 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2210780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2210780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2210780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores
   [junit4]   2> 2210781 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2210781 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1
   [junit4]   2> 2210781 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 2210782 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 2210782 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:59752_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 2210784 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2210784 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2210785 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 2210785 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 2210785 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 2210785 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 2210787 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59752",
   [junit4]   2>   "node_name":"127.0.0.1:59752_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2210787 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59752",
   [junit4]   2>   "node_name":"127.0.0.1:59752_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2210788 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 2210788 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2210790 INFO  (zkCallback-2432-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2211787 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2211787 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 2211787 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 2211787 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 2211787 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 2211788 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1'
   [junit4]   2> 2211788 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2211788 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2211789 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 2211792 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 2211796 WARN  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 2211797 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 2211812 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2211817 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2211879 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2211882 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 2211890 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 2211898 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2211900 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2211902 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2211903 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2211904 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 2211904 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2211904 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2211905 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data/]
   [junit4]   2> 2211905 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4ad80101
   [junit4]   2> 2211905 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data
   [junit4]   2> 2211905 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data/index/
   [junit4]   2> 2211905 WARN  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2211906 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data/index
   [junit4]   2> 2211906 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=36, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4045144742146216]
   [junit4]   2> 2211906 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@934c492 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@685c588),segFN=segments_1,generation=1}
   [junit4]   2> 2211906 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 2211907 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data/
   [junit4]   2> 2211907 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2211910 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2211910 WARN  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 2211912 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2211912 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2211913 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2211914 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2211916 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 2211916 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2211917 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2211917 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2211918 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2211918 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2211918 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=42, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=61.1064453125, floorSegmentMB=0.953125, forceMergeDeletesPctAllowed=29.561958091259484, segmentsPerTier=49.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 2211918 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@934c492 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@685c588),segFN=segments_1,generation=1}
   [junit4]   2> 2211918 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 2211918 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6bbff668[collection1] main]
   [junit4]   2> 2211919 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 2211919 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 2211919 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 2211920 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2211921 INFO  (searcherExecutor-6017-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6bbff668[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2211921 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 2211921 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 2211921 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 2211921 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1533430422197764096
   [junit4]   2> 2211924 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 3.0ms to seed version buckets with highest version 1533430422197764096
   [junit4]   2> 2211924 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 2211925 INFO  (coreLoadExecutor-6016-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 2211925 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:59752 collection:control_collection shard:shard1
   [junit4]   2> 2211926 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2211928 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95839401293905926-core_node1-n_0000000000
   [junit4]   2> 2211928 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 2211929 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2211929 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2211929 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:59752/collection1/
   [junit4]   2> 2211929 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2211929 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:59752/collection1/ has no replicas
   [junit4]   2> 2211929 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 2211930 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2211931 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95839401293905926-core_node1-n_0000000000
   [junit4]   2> 2211934 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:59752/collection1/ shard1
   [junit4]   2> 2211934 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:59752",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 2212036 INFO  (zkCallback-2432-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2212085 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:59752/collection1/ and leader is http://127.0.0.1:59752/collection1/
   [junit4]   2> 2212085 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:59752
   [junit4]   2> 2212085 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2212085 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 2212085 INFO  (coreZkRegister-6010-thread-1-processing-n:127.0.0.1:59752_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:59752_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2212086 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59752",
   [junit4]   2>   "node_name":"127.0.0.1:59752_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 2212087 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:59752",
   [junit4]   2>   "node_name":"127.0.0.1:59752_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2212188 INFO  (zkCallback-2432-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2212284 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2212284 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2212285 INFO  (zkCallback-2434-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@408451f4 name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2212285 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2212285 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2212285 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2212290 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2212290 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2212346 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/cores/collection1
   [junit4]   2> 2212347 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001
   [junit4]   2> 2212347 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 2212348 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ac7d460{/,null,AVAILABLE}
   [junit4]   2> 2212349 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.ServerConnector Started ServerConnector@18a78c88{HTTP/1.1,[http/1.1]}{127.0.0.1:46267}
   [junit4]   2> 2212349 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.e.j.s.Server Started @2214615ms
   [junit4]   2> 2212349 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=46267, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/cores}
   [junit4]   2> 2212349 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 2212349 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001'
   [junit4]   2> 2212350 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2212350 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2212350 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2212350 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2212351 INFO  (zkCallback-2435-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@974b456 name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2212351 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2212351 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2212355 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2212355 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/solr.xml
   [junit4]   2> 2212358 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/cores
   [junit4]   2> 2212358 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CoreContainer New CoreContainer 351112566
   [junit4]   2> 2212358 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001]
   [junit4]   2> 2212358 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/lib
   [junit4]   2> 2212359 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2212359 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2212360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2212360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2212360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:57646/solr
   [junit4]   2> 2212360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2212360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2212360 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2212361 INFO  (zkCallback-2438-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1ce2ddd6 name:ZooKeeperConnection Watcher:127.0.0.1:57646 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2212361 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2212361 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2212362 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$27/1828077232@6a2831d8
   [junit4]   2> 2212362 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2212363 INFO  (zkCallback-2439-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6095ac80 name:ZooKeeperConnection Watcher:127.0.0.1:57646/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2212363 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2212367 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2212368 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2212370 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2212370 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95839401293905930-127.0.0.1:46267_-n_0000000001
   [junit4]   2> 2212374 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95839401293905926-127.0.0.1:59752_-n_0000000000 to know if I could be the leader
   [junit4]   2> 2212374 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.ZkController Publish node=127.0.0.1:46267_ as DOWN
   [junit4]   2> 2212376 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:46267_"} current state version: 3
   [junit4]   2> 2212376 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:46267_
   [junit4]   2> 2212478 INFO  (zkCallback-2434-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2212479 INFO  (zkCallback-2432-thread-1-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2212479 INFO  (zkCallback-2439-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2213376 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46267_
   [junit4]   2> 2213377 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:46267_
   [junit4]   2> 2213379 INFO  (zkCallback-2439-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2213379 INFO  (zkCallback-2434-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2213379 INFO  (zkCallback-2432-thread-2-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2213379 INFO  (zkCallback-2434-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2213379 INFO  (zkCallback-2439-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2213381 INFO  (zkCallback-2432-thread-2-processing-n:127.0.0.1:59752_) [n:127.0.0.1:59752_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2213387 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2213387 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2213387 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/cores
   [junit4]   2> 2213388 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2213388 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_358C62A1A273FF2D-001/shard-1-001/cores/collection1
   [junit4]   2> 2213388 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 2213389 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 2213389 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[358C62A1A273FF2D]) [n:127.0.0.1:46267_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 2213389 INFO  (coreLoadExecutor-6027-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2213390 INFO  (coreLoadExecutor-6027-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2213390 INFO  (coreLoadExecutor-6027-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_ c:collection1   x:collection1] o.a.s.c.ZkController Collection collection1 not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 2213390 INFO  (coreLoadExecutor-6027-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 2213390 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46267",
   [junit4]   2>   "node_name":"127.0.0.1:46267_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 4
   [junit4]   2> 2213390 INFO  (coreLoadExecutor-6027-thread-1-processing-n:127.0.0.1:46267_) [n:127.0.0.1:46267_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [collection1]
   [junit4]   2> 2213390 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:46267",
   [junit4]   2>   "node_name":"127.0.0.1:46267_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2213391 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 2213391 INFO  (OverseerStateUpdate-95839401293905926-127.0.0.1:59752_-n_0000000000) [n:127.0.0.1:59752_    ] o.a.s.c.o.ReplicaMutator Assigni

[...truncated too long message...]

-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/icu/lucene-analyzers-icu-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J0-20160504_194315_416.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J0-20160504_194315_416.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 233 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 55 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/55/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:59307

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:59307
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:457)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
	at __randomizedtesting.SeedInfo.seed([C8145CEB159DF6BE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12500 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/init-core-data-001
   [junit4]   2> 1706562 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C8145CEB159DF6BE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1706562 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C8145CEB159DF6BE]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1706564 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1706565 INFO  (Thread-22087) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1706565 INFO  (Thread-22087) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1706665 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ZkTestServer start zk server on port:56141
   [junit4]   2> 1706665 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1706670 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706672 INFO  (zkCallback-1204-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@73dcb03 name:ZooKeeperConnection Watcher:127.0.0.1:56141 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706673 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706674 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1706674 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1706677 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1706683 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706692 INFO  (zkCallback-1205-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7e913cbb name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706692 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1706693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1706694 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1706695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1706696 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1706697 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1706697 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1706711 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1706711 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1706715 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1706715 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1706718 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1706718 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1706722 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1706722 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1706731 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1706731 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1706734 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1706734 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1706736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1706736 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1706737 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1706738 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1706739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1706739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1706740 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1706741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1706743 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1706743 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706744 INFO  (zkCallback-1206-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@527bb8de name:ZooKeeperConnection Watcher:127.0.0.1:56141 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706746 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706746 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1706746 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1706749 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1706755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706757 INFO  (zkCallback-1207-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@26b9412d name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706757 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706757 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1706758 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1706758 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1706761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1706761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 1706767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 1706767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 1706769 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1706769 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1706770 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1706770 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1706772 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1706773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1706775 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1706775 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 1706777 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1706777 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1706780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1706780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1706783 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1706783 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1706793 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1706794 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1706796 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1706796 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1706950 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1
   [junit4]   2> 1706952 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1706956 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@e0fa46d{/,null,AVAILABLE}
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4ed541c9{HTTP/1.1,[http/1.1]}{127.0.0.1:35045}
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.Server Started @1710103ms
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/tempDir-001/control/data, hostContext=/, hostPort=35045, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores}
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001'
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1706957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1706964 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706971 INFO  (zkCallback-1208-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@253e2529 name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706971 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706971 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1706972 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1706972 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/solr.xml
   [junit4]   2> 1706977 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores
   [junit4]   2> 1706977 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CoreContainer New CoreContainer 170742918
   [junit4]   2> 1706977 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001]
   [junit4]   2> 1706977 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/lib
   [junit4]   2> 1706978 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1706979 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1706980 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1706980 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1706980 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56141/solr
   [junit4]   2> 1706980 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1706980 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1706983 INFO  (zkCallback-1211-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5be943c0 name:ZooKeeperConnection Watcher:127.0.0.1:56141 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706987 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706987 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706987 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1706988 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$47/1525438528@2718f158
   [junit4]   2> 1706988 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1706989 INFO  (zkCallback-1212-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@69187537 name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1706989 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1706990 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1706992 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1706995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1706997 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1706999 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1707001 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1707002 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1707004 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1707005 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1707005 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1707006 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1707008 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1707009 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1707010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1707010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95834174887100422-127.0.0.1:35045_-n_0000000000
   [junit4]   2> 1707011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35045_
   [junit4]   2> 1707011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1707012 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer Overseer (id=95834174887100422-127.0.0.1:35045_-n_0000000000) starting
   [junit4]   2> 1707013 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1707018 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1707019 INFO  (OverseerCollectionConfigSetProcessor-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1707019 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1707019 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35045_
   [junit4]   2> 1707021 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1707021 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:35045_
   [junit4]   2> 1707022 INFO  (zkCallback-1212-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1707022 INFO  (zkCallback-1212-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1707037 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1707037 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1707038 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores
   [junit4]   2> 1707039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1707039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1
   [junit4]   2> 1707039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1707043 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1707043 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:35045_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1707043 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1707043 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1707044 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1707044 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 1707045 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35045",
   [junit4]   2>   "node_name":"127.0.0.1:35045_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1707045 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 1707045 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35045",
   [junit4]   2>   "node_name":"127.0.0.1:35045_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1707046 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1707046 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1707046 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1707047 INFO  (zkCallback-1212-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1708046 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1708046 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1708047 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1708047 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1708048 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1708048 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1'
   [junit4]   2> 1708048 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1708048 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1708050 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1708054 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1708058 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1708066 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1708071 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1708152 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1708157 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1708158 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1708165 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1708167 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1708169 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1708170 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1708171 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1708171 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1708171 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1708171 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data/]
   [junit4]   2> 1708172 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@5ad4d3f6
   [junit4]   2> 1708172 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data
   [junit4]   2> 1708172 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1708172 WARN  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1708173 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data/index
   [junit4]   2> 1708173 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=44, maxMergeAtOnceExplicit=25, maxMergedSegmentMB=27.8388671875, floorSegmentMB=1.16015625, forceMergeDeletesPctAllowed=2.535092880785077, segmentsPerTier=32.0, maxCFSSegmentSizeMB=0.3173828125, noCFSRatio=1.0
   [junit4]   2> 1708173 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@76fbc153 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ebfa52e),segFN=segments_1,generation=1}
   [junit4]   2> 1708173 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1708174 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data/
   [junit4]   2> 1708174 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1708177 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1708177 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1708177 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1708177 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1708177 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1708177 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1708178 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1708178 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1708178 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1708178 WARN  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1708180 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1708180 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1708181 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1708182 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1708185 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1708185 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1708185 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1708185 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1708186 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1708186 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1708186 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=10, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7771397645231398]
   [junit4]   2> 1708186 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@76fbc153 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@ebfa52e),segFN=segments_1,generation=1}
   [junit4]   2> 1708186 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1708187 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@632f1b73[collection1] main]
   [junit4]   2> 1708187 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1708188 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1708189 INFO  (searcherExecutor-3922-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@632f1b73[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1708189 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1708189 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1708189 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1708189 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1533346799881814016
   [junit4]   2> 1708191 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1533346799881814016
   [junit4]   2> 1708191 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1708192 INFO  (coreLoadExecutor-3921-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1708192 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:35045 collection:control_collection shard:shard1
   [junit4]   2> 1708195 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1708198 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95834174887100422-core_node1-n_0000000000
   [junit4]   2> 1708198 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1708199 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1708199 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1708199 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:35045/collection1/
   [junit4]   2> 1708199 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1708199 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:35045/collection1/ has no replicas
   [junit4]   2> 1708200 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1708200 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1708202 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95834174887100422-core_node1-n_0000000000
   [junit4]   2> 1708203 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:35045/collection1/ shard1
   [junit4]   2> 1708204 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:35045",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1708306 INFO  (zkCallback-1212-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1708354 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:35045/collection1/ and leader is http://127.0.0.1:35045/collection1/
   [junit4]   2> 1708354 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:35045
   [junit4]   2> 1708354 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1708354 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1708354 INFO  (coreZkRegister-3915-thread-1-processing-n:127.0.0.1:35045_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:35045_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1708356 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35045",
   [junit4]   2>   "node_name":"127.0.0.1:35045_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1708356 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.0.0.1:35045_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:35045",
   [junit4]   2>   "node_name":"127.0.0.1:35045_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1708458 INFO  (zkCallback-1212-thread-1-processing-n:127.0.0.1:35045_) [n:127.0.0.1:35045_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1708544 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:56141/solr
   [junit4]   2> 1708545 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1708545 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1708547 INFO  (zkCallback-1214-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6459d11a name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1708547 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1708547 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1708547 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1708549 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1708549 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1708638 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001/cores/collection1
   [junit4]   2> 1708639 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001
   [junit4]   2> 1708639 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1708641 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1bb09465{/,null,AVAILABLE}
   [junit4]   2> 1708641 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.ServerConnector Started ServerConnector@1c4c9441{HTTP/1.1,[http/1.1]}{127.0.0.1:59307}
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.e.j.s.Server Started @1711788ms
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=59307, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001/cores}
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001'
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1708642 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1708643 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1708644 INFO  (zkCallback-1215-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@32258027 name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1708644 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1708644 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1708645 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1708645 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001/solr.xml
   [junit4]   2> 1708649 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001/cores
   [junit4]   2> 1708650 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CoreContainer New CoreContainer 1975698588
   [junit4]   2> 1708650 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001]
   [junit4]   2> 1708650 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C8145CEB159DF6BE-001/shard-1-001/lib
   [junit4]   2> 1708650 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1708651 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1708651 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1708651 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1708651 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56141/solr
   [junit4]   2> 1708652 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1708652 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1708652 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1708653 INFO  (zkCallback-1218-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@567f1878 name:ZooKeeperConnection Watcher:127.0.0.1:56141 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1708653 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1708653 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1708654 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$47/1525438528@794a49e1
   [junit4]   2> 1708654 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1708655 INFO  (zkCallback-1219-thread-1-processing-n:127.0.0.1:59307_) [n:127.0.0.1:59307_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2bf473de name:ZooKeeperConnection Watcher:127.0.0.1:56141/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1708655 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1708661 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1708662 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1708664 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1708664 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95834174887100426-127.0.0.1:59307_-n_0000000001
   [junit4]   2> 1708664 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95834174887100422-127.0.0.1:35045_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1708665 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[C8145CEB159DF6BE]) [n:127.0.0.1:59307_    ] o.a.s.c.ZkController Publish node=127.0.0.1:59307_ as DOWN
   [junit4]   2> 1708666 INFO  (OverseerStateUpdate-95834174887100422-127.0.0.1:35045_-n_0000000000) [n:127.

[...truncated too long message...]

g-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/icu/lucene-analyzers-icu-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J0-20160503_214229_802.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J0-20160503_214229_802.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 272 minutes 7 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 54 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/54/

All tests passed

Build Log:
[...truncated 12524 lines...]
   [junit4] JVM J2: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J2-20160501_230636_014.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid11594.hprof ...
   [junit4] Heap dump file created [615402481 bytes in 2.732 secs]
   [junit4] <<< JVM J2: EOF ----

   [junit4] JVM J2: stderr was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J2-20160501_230636_014.syserr
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) ----
   [junit4] 
   [junit4] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "SessionTracker"
   [junit4] WARN: Unhandled exception in event serialization. -> java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] <<< JVM J2: EOF ----

[...truncated 339 lines...]
   [junit4] ERROR: JVM J2 ended with an exception, command line: /x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/jre/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps -ea -esa -Dtests.prefix=tests -Dtests.seed=DFDAC0F3EAFBB7BB -Xmx512M -Dtests.iters= -Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random -Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.luceneMatchVersion=6.1.0 -Dtests.cleanthreads=perClass -Djava.util.logging.config.file=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/tools/junit4/logging.properties -Dtests.nightly=true -Dtests.weekly=false -Dtests.monster=false -Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=2 -DtempDir=./temp -Djava.io.tmpdir=./temp -Djunit4.tempDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp -Dcommon.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene -Dclover.db.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/clover/db -Djava.security.policy=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/tools/junit4/solr-tests.policy -Dtests.LUCENE_VERSION=6.1.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 -Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory -Djava.awt.headless=true -Djdk.map.althashing.threshold=0 -Djunit4.childvm.cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false -Dtests.filterstacks=true -Djava.security.manager=org.apache.lucene.util.TestSecurityManager -Dfile.encoding=ISO-8859-1 -classpath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/test-framework/lib/junit4-ant-2.3.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/test-framework/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/codecs/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-solrj/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/common/lucene-analyzers-common-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/kuromoji/lucene-analyzers-kuromoji-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/phonetic/lucene-analyzers-phonetic-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/codecs/lucene-codecs-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/backward-codecs/lucene-backward-codecs-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/highlighter/lucene-highlighter-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/memory/lucene-memory-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/misc/lucene-misc-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/spatial-extras/lucene-spatial-extras-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/expressions/lucene-expressions-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/suggest/lucene-suggest-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/grouping/lucene-grouping-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queries/lucene-queries-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queryparser/lucene-queryparser-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/join/lucene-join-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/sandbox/lucene-sandbox-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/classification/lucene-classification-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/antlr4-runtime-4.5.1-1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/asm-5.0.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/asm-commons-5.0.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/caffeine-1.0.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-cli-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-codec-1.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-configuration-1.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-exec-1.3.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-fileupload-1.2.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/commons-lang-2.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/dom4j-1.6.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/guava-14.0.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/hadoop-annotations-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/hadoop-auth-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/hadoop-common-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/hadoop-hdfs-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/hppc-0.7.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/htrace-core-3.0.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/jackson-core-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/jackson-dataformat-smile-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/joda-time-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/log4j-1.2.17.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/org.restlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/org.restlet.ext.servlet-2.3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/protobuf-java-2.5.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/slf4j-log4j12-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/spatial4j-0.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/lib/t-digest-3.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/commons-io-2.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/httpclient-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/httpcore-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/httpmime-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/jcl-over-slf4j-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/noggit-0.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/slf4j-api-1.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/stax2-api-3.1.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/woodstox-core-asl-4.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/solrj/lib/zookeeper-3.4.6.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/javax.servlet-api-3.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-continuation-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-deploy-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-http-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-io-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-jmx-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-rewrite-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-security-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-server-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-servlet-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-servlets-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-util-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-webapp-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/lib/jetty-xml-9.3.8.v20160314.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/example/example-DIH/solr/db/lib/derby-10.9.1.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/example/example-DIH/solr/db/lib/hsqldb-1.8.0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/test-framework/lib/junit-4.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/test-framework/lib/randomizedtesting-runner-2.3.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/antlr-2.7.7.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/antlr4-runtime-4.5.1-1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-core-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-core-api-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-core-avl-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-core-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-i18n-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptor-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-admin-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-authn-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-authz-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-changelog-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/icu/lucene-analyzers-icu-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J2-20160501_230636_014.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J2-20160501_230636_014.suites -stdin
   [junit4] ERROR: JVM J2 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 211 minutes 12 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 53 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/53/

2 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:35756/q_s/nl

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:35756/q_s/nl
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:457)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:59625/htp

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:59625/htp
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:457)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more




Build Log:
[...truncated 11549 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/init-core-data-001
   [junit4]   2> 1053049 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1053050 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /q_s/nl
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1053101 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1053107 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1053109 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1053118 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_36876_hdfs____.mh7mgy/webapp
   [junit4]   2> 1053198 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1053535 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36876
   [junit4]   2> 1053614 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1053615 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1053627 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40882_datanode____.622wv7/webapp
   [junit4]   2> 1053712 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1054109 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40882
   [junit4]   2> 1054177 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1054178 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1054193 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_41105_datanode____5d7a2o/webapp
   [junit4]   2> 1054298 INFO  (IPC Server handler 0 on 46442) [    ] BlockStateChange BLOCK* processReport: from storage DS-196ac04a-ca8c-409d-9c07-cb11eef0f31b node DatanodeRegistration(127.0.0.1, datanodeUuid=81d99c3d-719d-4d6b-9066-1db4acdfc845, infoPort=40882, ipcPort=54195, storageInfo=lv=-56;cid=testClusterID;nsid=1522856014;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1054298 INFO  (IPC Server handler 0 on 46442) [    ] BlockStateChange BLOCK* processReport: from storage DS-0740bc36-0b7e-49cc-8a35-92cc8235e0e4 node DatanodeRegistration(127.0.0.1, datanodeUuid=81d99c3d-719d-4d6b-9066-1db4acdfc845, infoPort=40882, ipcPort=54195, storageInfo=lv=-56;cid=testClusterID;nsid=1522856014;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1054318 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1054756 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41105
   [junit4]   2> 1054821 INFO  (IPC Server handler 1 on 46442) [    ] BlockStateChange BLOCK* processReport: from storage DS-3ccc41be-04d3-460e-987d-c074591f2af0 node DatanodeRegistration(127.0.0.1, datanodeUuid=2e289dbb-da58-4768-a06e-ba3f20b8f7ef, infoPort=41105, ipcPort=35063, storageInfo=lv=-56;cid=testClusterID;nsid=1522856014;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1054821 INFO  (IPC Server handler 1 on 46442) [    ] BlockStateChange BLOCK* processReport: from storage DS-5d914457-994b-4549-937c-f5c65ba8391b node DatanodeRegistration(127.0.0.1, datanodeUuid=2e289dbb-da58-4768-a06e-ba3f20b8f7ef, infoPort=41105, ipcPort=35063, storageInfo=lv=-56;cid=testClusterID;nsid=1522856014;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1054923 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1054923 INFO  (Thread-7579) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1054923 INFO  (Thread-7579) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1055023 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.ZkTestServer start zk server on port:46819
   [junit4]   2> 1055023 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1055024 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055026 INFO  (zkCallback-1657-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d37ab42 name:ZooKeeperConnection Watcher:127.0.0.1:46819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055026 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055026 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1055026 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1055027 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1055028 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055029 INFO  (zkCallback-1658-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@751baaf6 name:ZooKeeperConnection Watcher:127.0.0.1:46819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055029 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055029 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1055029 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1055031 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1055032 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1055033 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1055034 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1055034 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1055036 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1055036 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1055037 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1055037 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1055038 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1055038 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1055039 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1055039 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1055039 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1055039 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1055040 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1055040 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1055041 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1055041 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1055043 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1055043 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1055045 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1055045 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1055046 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1055046 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1055049 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1055049 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055050 INFO  (zkCallback-1659-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3f46d1c8 name:ZooKeeperConnection Watcher:127.0.0.1:46819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055050 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055050 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1055050 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1055051 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1055051 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055051 INFO  (zkCallback-1660-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@27f3e05d name:ZooKeeperConnection Watcher:127.0.0.1:46819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055051 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055051 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1055052 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1055052 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1055053 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1055053 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 1055054 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 1055054 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 1055055 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1055055 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1055056 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1055056 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1055057 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1055057 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1055057 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1055057 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 1055058 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1055058 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1055059 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1055059 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1055060 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1055060 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1055061 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1055061 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1055062 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1055062 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1055120 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores/collection1
   [junit4]   2> 1055121 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1055122 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@703be2b6{/q_s/nl,null,AVAILABLE}
   [junit4]   2> 1055122 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.e.j.s.ServerConnector Started ServerConnector@23ba2398{HTTP/1.1,[http/1.1]}{127.0.0.1:56494}
   [junit4]   2> 1055122 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.e.j.s.Server Started @1057204ms
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:46442/hdfs__localhost_46442__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001_tempDir-002_control_data, hostContext=/q_s/nl, hostPort=56494, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores}
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001'
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1055123 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055125 INFO  (zkCallback-1661-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3131617c name:ZooKeeperConnection Watcher:127.0.0.1:46819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055125 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055125 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1055125 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1055126 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/solr.xml
   [junit4]   2> 1055129 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores
   [junit4]   2> 1055129 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.CoreContainer New CoreContainer 1310770332
   [junit4]   2> 1055129 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001]
   [junit4]   2> 1055129 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/lib
   [junit4]   2> 1055129 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1055130 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1055130 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1055130 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1055131 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:46819/solr
   [junit4]   2> 1055131 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1055131 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1055131 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055133 INFO  (zkCallback-1664-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4025df80 name:ZooKeeperConnection Watcher:127.0.0.1:46819 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055134 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055134 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1055135 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$32/702459987@42918664
   [junit4]   2> 1055135 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1055136 INFO  (zkCallback-1665-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@68ba7591 name:ZooKeeperConnection Watcher:127.0.0.1:46819/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1055137 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1055138 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1055138 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1055140 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1055141 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1055142 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1055143 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1055143 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1055144 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1055145 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1055145 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1055146 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1055146 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1055147 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1055148 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1055148 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000
   [junit4]   2> 1055148 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:56494_q_s%2Fnl
   [junit4]   2> 1055148 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1055149 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer Overseer (id=95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) starting
   [junit4]   2> 1055150 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1055153 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1055153 INFO  (OverseerCollectionConfigSetProcessor-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1055153 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1055154 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:56494_q_s%2Fnl
   [junit4]   2> 1055154 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:56494_q_s%2Fnl
   [junit4]   2> 1055155 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1055155 INFO  (zkCallback-1665-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1055155 INFO  (zkCallback-1665-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1055174 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1055174 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1055174 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores
   [junit4]   2> 1055175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1055175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores/collection1
   [junit4]   2> 1055175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1055175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1055175 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[54A59967B5CDEEA1]) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1055176 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1055176 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1055176 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1055176 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 1055176 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56494/q_s/nl",
   [junit4]   2>   "node_name":"127.0.0.1:56494_q_s%2Fnl",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1055177 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56494/q_s/nl",
   [junit4]   2>   "node_name":"127.0.0.1:56494_q_s%2Fnl",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1055177 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 1055177 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1055177 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1055177 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1055178 INFO  (zkCallback-1665-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1056178 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1056178 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1056179 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1056179 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1056179 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1056179 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores/collection1'
   [junit4]   2> 1056179 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1056179 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1056180 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1056184 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1056198 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1056204 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1056209 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1056283 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1056287 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1056288 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1056296 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1056298 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1056300 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1056300 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1056301 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1056301 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1056302 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 1056302 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:46442/solr_hdfs_home
   [junit4]   2> 1056302 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1056302 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001/control-001/cores/collection1], dataDir=[hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1056302 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1a54c8a2
   [junit4]   2> 1056302 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1056316 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1056318 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 1056325 WARN  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:46442/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1056325 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1056335 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1056341 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=7.255859375, floorSegmentMB=0.7236328125, forceMergeDeletesPctAllowed=14.846088454046617, segmentsPerTier=31.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.4331034022008823
   [junit4]   2> 1056367 INFO  (IPC Server handler 1 on 46442) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36831 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-5d914457-994b-4549-937c-f5c65ba8391b:NORMAL:127.0.0.1:39142|RBW], ReplicaUnderConstruction[[DISK]DS-196ac04a-ca8c-409d-9c07-cb11eef0f31b:NORMAL:127.0.0.1:36831|RBW]]} size 0
   [junit4]   2> 1056368 INFO  (IPC Server handler 7 on 46442) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:39142 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-196ac04a-ca8c-409d-9c07-cb11eef0f31b:NORMAL:127.0.0.1:36831|RBW], ReplicaUnderConstruction[[DISK]DS-3ccc41be-04d3-460e-987d-c074591f2af0:NORMAL:127.0.0.1:39142|FINALIZED]]} size 0
   [junit4]   2> 1056370 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6311ab85; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1056370 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1056379 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 1056381 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1056381 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1056381 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1056381 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1056382 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1056382 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1056382 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1056382 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1056382 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1056382 WARN  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1056383 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1056384 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1056385 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1056385 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1056387 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1056387 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1056387 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1056387 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1056388 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1056397 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1056397 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1056398 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=5, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3464881846070783]
   [junit4]   2> 1056404 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6311ab85; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1056404 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1056404 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@ed07d90[collection1] main]
   [junit4]   2> 1056404 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1056405 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1056405 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1056405 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1056405 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1056405 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1056405 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1056406 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1056406 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1056406 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1056407 INFO  (searcherExecutor-4012-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@ed07d90[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1056407 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1056407 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1056407 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1056407 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1533084198032637952
   [junit4]   2> 1056409 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1533084198032637952
   [junit4]   2> 1056409 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1056409 INFO  (coreLoadExecutor-4011-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1056409 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:56494/q_s/nl collection:control_collection shard:shard1
   [junit4]   2> 1056410 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1056412 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95817762279849990-core_node1-n_0000000000
   [junit4]   2> 1056412 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1056413 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1056413 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1056413 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:56494/q_s/nl/collection1/
   [junit4]   2> 1056413 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1056413 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:56494/q_s/nl/collection1/ has no replicas
   [junit4]   2> 1056413 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1056414 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1056415 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95817762279849990-core_node1-n_0000000000
   [junit4]   2> 1056416 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:56494/q_s/nl/collection1/ shard1
   [junit4]   2> 1056417 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:56494/q_s/nl",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1056520 INFO  (zkCallback-1665-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1056567 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:56494/q_s/nl/collection1/ and leader is http://127.0.0.1:56494/q_s/nl/collection1/
   [junit4]   2> 1056567 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:56494/q_s/nl
   [junit4]   2> 1056567 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1056567 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1056567 INFO  (coreZkRegister-4005-thread-1-processing-n:127.0.0.1:56494_q_s%2Fnl x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:56494_q_s%2Fnl c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1056569 INFO  (OverseerStateUpdate-95817762279849990-127.0.0.1:56494_q_s%2Fnl-n_0000000000) [n:127.0.0.1:56494_q_s%2Fnl    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "dataDir":"hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:56494/q_s/nl",
   [junit4]   2>   "node_name":"127.0.0.1:56494_q_s%2Fnl",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "ulogDir":"hdfs://localhost:46442/solr_hdfs_home/control_collection/core_node1/data/tlog"} current state version: 2
   [junit4]   2> 1056569 INFO  (OverseerSt

[...truncated too long message...]

.CloudSolrClient.request(CloudSolrClient.java:898)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:987)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33921/htp: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
   [junit4]    > 	... 7 moreThrowable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25232, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.RuntimeException: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:33921/htp, http://127.0.0.1:57272/htp, http://127.0.0.1:59625/htp, http://127.0.0.1:40557/htp, http://127.0.0.1:58902/htp]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([54A59967B5CDEEA1]:0)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:997)
   [junit4]    > Caused by: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[http://127.0.0.1:33921/htp, http://127.0.0.1:57272/htp, http://127.0.0.1:59625/htp, http://127.0.0.1:40557/htp, http://127.0.0.1:58902/htp]
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:384)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:987)
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33921/htp: KeeperErrorCode = Session expired for /overseer/collection-queue-work/qnr-
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
   [junit4]    > 	... 7 more
   [junit4]   2> 2881334 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[54A59967B5CDEEA1]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_54A59967B5CDEEA1-001
   [junit4]   2> May 01, 2016 1:07:29 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=FSTOrd50, range_facet_l_dv=PostingsFormat(name=LuceneFixedGap), _version_=FSTOrd50, multiDefault=PostingsFormat(name=MockRandom), a_t=FSTOrd50, intDefault=FSTOrd50, a_i1=FSTOrd50, id=PostingsFormat(name=LuceneFixedGap), range_facet_i_dv=FSTOrd50, text=BlockTreeOrds(blocksize=128), range_facet_l=FSTOrd50, timestamp=FSTOrd50}, docValues:{range_facet_l_dv=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Asserting), timestamp=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=453, maxMBSortInHeap=5.287367129123488, sim=ClassicSimilarity, locale=pl, timezone=BET
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=161921336,total=477626368
   [junit4]   2> NOTE: All tests run in this JVM: [TestFreeTextSuggestions, TestSolrQueryResponse, TestConfigSets, TestReloadAndDeleteDocs, TestSolrDynamicMBean, TestCustomSort, FacetPivotSmallTest, TestCryptoKeys, TestLFUCache, CloneFieldUpdateProcessorFactoryTest, TestUpdate, TolerantUpdateProcessorTest, TestObjectReleaseTracker, TestDynamicFieldCollectionResource, CollectionsAPIAsyncDistributedZkTest, FastVectorHighlighterTest, TestSortByMinMaxFunction, TestSolrConfigHandlerCloud, SpatialRPTFieldTypeTest, CollectionReloadTest, CloudMLTQParserTest, TestConfigSetsAPI, TestManagedSynonymFilterFactory, NumericFieldsTest, SpatialFilterTest, TestWordDelimiterFilterFactory, SolrIndexSplitterTest, ParsingFieldUpdateProcessorsTest, TestQueryTypes, TestLMJelinekMercerSimilarityFactory, SortByFunctionTest, ZkCLITest, TestExceedMaxTermLength, TestBlobHandler, TestPartialUpdateDeduplication, TestMiniSolrCloudCluster, TestQuerySenderListener, TestHashPartitioner, AsyncMigrateRouteKeyTest, SpellCheckCollatorTest, TestReRankQParserPlugin, AddSchemaFieldsUpdateProcessorFactoryTest, DistributedQueryComponentCustomSortTest, RecoveryAfterSoftCommitTest, ZkControllerTest, TestUseDocValuesAsStored, TestDFRSimilarityFactory, HttpPartitionTest, CustomCollectionTest, TestEmbeddedSolrServerConstructors, TestPivotHelperCode, TestCloudInspectUtil, TermVectorComponentTest, OpenExchangeRatesOrgProviderTest, TestDistribIDF, MBeansHandlerTest, TestSha256AuthenticationProvider, TestSchemaNameResource, WordBreakSolrSpellCheckerTest, HdfsWriteToMultipleCollectionsTest, SolrXmlInZkTest, TestSimpleTrackingShardHandler, SpellPossibilityIteratorTest, BlockJoinFacetRandomTest, TestReqParamsAPI, TestPerFieldSimilarityWithDefaultOverride, ShardRoutingCustomTest, SchemaVersionSpecificBehaviorTest, DistributedIntervalFacetingTest, TestSizeLimitedDistributedMap, DOMUtilTest, ShardRoutingTest, HdfsCollectionsAPIDistributedZkTest, SolrPluginUtilsTest, SpellingQueryConverterTest, CSVRequestHandlerTest, SoftAutoCommitTest, TestBM25SimilarityFactory, TestSolrCoreProperties, DistributedExpandComponentTest, TriLevelCompositeIdRoutingTest, SolrIndexConfigTest, TestTolerantUpdateProcessorCloud, HdfsRecoveryZkTest, MinimalSchemaTest, SharedFSAutoReplicaFailoverUtilsTest, ConvertedLegacyTest, TestAuthorizationFramework, TestDistributedGrouping, FileUtilsTest, TestQuerySenderNoQuery, SimplePostToolTest, TestHashQParserPlugin, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed [288/592 (2!)] on J1 in 757.94s, 1 test, 1 error <<< FAILURES!

[...truncated 947 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J1-20160501_001926_019.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid20119.hprof ...
   [junit4] Heap dump file created [610644242 bytes in 6.399 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 23 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 592 suites (10 ignored), 2399 tests, 2 errors, 31 ignored (16 assumptions) [seed: 54A59967B5CDEEA1]

Total time: 211 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 52 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/52/

3 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=1094, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1094, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
	at __randomizedtesting.SeedInfo.seed([D87C2A1E874D0DFE:502815C429B16006]:0)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:55263: collection already exists: awholynewstresscollection_collection4_5
	at __randomizedtesting.SeedInfo.seed([D87C2A1E874D0DFE]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:987)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud

Error Message:
Could not find collection:test_col

Stack Trace:
java.lang.AssertionError: Could not find collection:test_col
	at __randomizedtesting.SeedInfo.seed([D87C2A1E874D0DFE]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:150)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorRandomCloud.java:135)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Sat Apr 30 07:51:23 ICT 2016

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Sat Apr 30 07:51:23 ICT 2016
	at __randomizedtesting.SeedInfo.seed([D87C2A1E874D0DFE:3D72AD88265644D]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1428)
	at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:780)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11310 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/init-core-data-001
   [junit4]   2> 65590 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[D87C2A1E874D0DFE]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 65590 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[D87C2A1E874D0DFE]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 65594 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 65595 INFO  (Thread-344) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 65595 INFO  (Thread-344) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 65695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ZkTestServer start zk server on port:39223
   [junit4]   2> 65695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 65696 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 65714 INFO  (zkCallback-60-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1c45fb97 name:ZooKeeperConnection Watcher:127.0.0.1:39223 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 65714 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 65715 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 65718 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 65730 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 65735 INFO  (zkCallback-61-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@327c3eb4 name:ZooKeeperConnection Watcher:127.0.0.1:39223/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 65735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 65735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 65737 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 65739 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 65740 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 65741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 65741 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 65745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 65745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 65750 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 65751 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 65753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 65753 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 65755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 65755 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 65756 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 65756 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 65757 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 65758 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 65759 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 65759 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 65761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 65761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 65765 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 65765 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 65767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 65767 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 65878 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1
   [junit4]   2> 65880 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 65882 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@23bfc24b{/,null,AVAILABLE}
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.ServerConnector Started ServerConnector@7ec9621e{HTTP/1.1,[http/1.1]}{127.0.0.1:53731}
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.Server Started @68824ms
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/tempDir-001/control/data, hostContext=/, hostPort=53731, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores}
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001'
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 65885 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 65886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 65886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 65889 INFO  (zkCallback-62-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@222ef9fb name:ZooKeeperConnection Watcher:127.0.0.1:39223/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65889 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 65890 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 65892 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 65892 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/solr.xml
   [junit4]   2> 65899 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores
   [junit4]   2> 65899 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CoreContainer New CoreContainer 3858867
   [junit4]   2> 65899 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001]
   [junit4]   2> 65900 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/lib
   [junit4]   2> 65900 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 65901 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 65902 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 65902 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 65902 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39223/solr
   [junit4]   2> 65902 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 65902 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 65903 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 65906 INFO  (zkCallback-65-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3ddf3a9b name:ZooKeeperConnection Watcher:127.0.0.1:39223 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65906 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 65907 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 65909 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$38/887802213@2304a10c
   [junit4]   2> 65910 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 65912 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4c9e88c8 name:ZooKeeperConnection Watcher:127.0.0.1:39223/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 65912 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 65915 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 65917 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 65919 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 65921 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 65922 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 65924 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 65927 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 65928 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 65929 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 65930 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 65931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 65932 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 65933 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 65934 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 65935 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95812075095588868-127.0.0.1:53731_-n_0000000000
   [junit4]   2> 65935 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53731_
   [junit4]   2> 65935 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 65936 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer Overseer (id=95812075095588868-127.0.0.1:53731_-n_0000000000) starting
   [junit4]   2> 65938 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 65943 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 65944 INFO  (OverseerCollectionConfigSetProcessor-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 65944 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 65944 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53731_
   [junit4]   2> 65945 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:53731_
   [junit4]   2> 65946 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 65946 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 65949 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 65971 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 65971 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 65972 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores
   [junit4]   2> 65973 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 65973 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1
   [junit4]   2> 65973 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 65974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 65974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:53731_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 65974 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 65974 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 65976 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 65976 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 65976 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:53731",
   [junit4]   2>   "node_name":"127.0.0.1:53731_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 65977 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:53731",
   [junit4]   2>   "node_name":"127.0.0.1:53731_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 65977 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 65977 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 65978 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 65979 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 65980 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66979 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 66980 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 66980 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 66980 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 66981 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 66981 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1'
   [junit4]   2> 66981 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 66981 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 66983 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 66996 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 67002 WARN  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 67003 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 67012 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 67018 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 67111 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 67117 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 67118 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 67132 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 67134 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 67136 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 67137 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 67142 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 67142 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 67143 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 67143 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data/]
   [junit4]   2> 67143 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3a7ad154
   [junit4]   2> 67144 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data
   [junit4]   2> 67145 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data/index/
   [junit4]   2> 67145 WARN  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 67145 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data/index
   [junit4]   2> 67145 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=6, maxMergedSegmentMB=77.5712890625, floorSegmentMB=1.3564453125, forceMergeDeletesPctAllowed=8.889765143520206, segmentsPerTier=16.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.7228485563731526
   [junit4]   2> 67146 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7ecab59b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c7ce6fe),segFN=segments_1,generation=1}
   [junit4]   2> 67146 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 67148 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data/
   [junit4]   2> 67149 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 67152 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 67152 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 67152 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 67152 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 67153 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 67153 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 67153 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 67153 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 67153 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 67153 WARN  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 67155 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 67156 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 67157 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 67158 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 67162 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 67162 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 67162 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 67162 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 67163 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 67163 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 67165 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.8426334006169524]
   [junit4]   2> 67165 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@7ecab59b lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@4c7ce6fe),segFN=segments_1,generation=1}
   [junit4]   2> 67165 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 67165 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@30712c90[collection1] main]
   [junit4]   2> 67165 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 67168 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 67168 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 67168 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 67169 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 67169 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 67169 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 67169 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 67169 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 67169 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 67170 INFO  (searcherExecutor-293-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@30712c90[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 67171 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 67171 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 67171 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 67171 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1532993203175686144
   [junit4]   2> 67172 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1532993203175686144
   [junit4]   2> 67172 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 67173 INFO  (coreLoadExecutor-292-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 67173 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:53731 collection:control_collection shard:shard1
   [junit4]   2> 67174 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 67178 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95812075095588868-core_node1-n_0000000000
   [junit4]   2> 67179 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 67181 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 67181 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 67181 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53731/collection1/
   [junit4]   2> 67181 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 67181 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:53731/collection1/ has no replicas
   [junit4]   2> 67182 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 67184 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95812075095588868-core_node1-n_0000000000
   [junit4]   2> 67184 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 67188 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:53731/collection1/ shard1
   [junit4]   2> 67188 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:53731",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 67291 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 67339 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:53731/collection1/ and leader is http://127.0.0.1:53731/collection1/
   [junit4]   2> 67339 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53731
   [junit4]   2> 67339 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 67340 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 67340 INFO  (coreZkRegister-286-thread-1-processing-n:127.0.0.1:53731_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53731_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 67342 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:53731",
   [junit4]   2>   "node_name":"127.0.0.1:53731_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 67343 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:53731",
   [junit4]   2>   "node_name":"127.0.0.1:53731_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 67446 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 67476 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 67477 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 67479 INFO  (zkCallback-68-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7c3d14d8 name:ZooKeeperConnection Watcher:127.0.0.1:39223/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67479 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 67479 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 67479 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 67485 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67486 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 67595 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/cores/collection1
   [junit4]   2> 67595 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001
   [junit4]   2> 67596 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 67598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2f928fd0{/,null,AVAILABLE}
   [junit4]   2> 67598 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.ServerConnector Started ServerConnector@48131c0d{HTTP/1.1,[http/1.1]}{127.0.0.1:40003}
   [junit4]   2> 67599 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.e.j.s.Server Started @70538ms
   [junit4]   2> 67599 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=40003, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/cores}
   [junit4]   2> 67599 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 67599 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001'
   [junit4]   2> 67599 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 67599 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 67600 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 67600 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 67606 INFO  (zkCallback-69-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2930bb33 name:ZooKeeperConnection Watcher:127.0.0.1:39223/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67606 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 67606 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 67610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 67610 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/solr.xml
   [junit4]   2> 67615 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/cores
   [junit4]   2> 67615 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CoreContainer New CoreContainer 1650000496
   [junit4]   2> 67615 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001]
   [junit4]   2> 67616 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/lib
   [junit4]   2> 67616 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 67617 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 67618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 67618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 67618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39223/solr
   [junit4]   2> 67618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 67618 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 67623 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 67628 INFO  (zkCallback-72-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@42eff37a name:ZooKeeperConnection Watcher:127.0.0.1:39223 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67629 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 67630 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 67631 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$38/887802213@6c184750
   [junit4]   2> 67636 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 67643 INFO  (zkCallback-73-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6852de2 name:ZooKeeperConnection Watcher:127.0.0.1:39223/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 67643 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 67649 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 67650 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 67652 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 67652 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95812075095588872-127.0.0.1:40003_-n_0000000001
   [junit4]   2> 67653 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95812075095588868-127.0.0.1:53731_-n_0000000000 to know if I could be the leader
   [junit4]   2> 67653 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.ZkController Publish node=127.0.0.1:40003_ as DOWN
   [junit4]   2> 67655 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:40003_"} current state version: 3
   [junit4]   2> 67655 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:40003_
   [junit4]   2> 67758 INFO  (zkCallback-68-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 67758 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 67758 INFO  (zkCallback-73-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 68655 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40003_
   [junit4]   2> 68656 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:40003_
   [junit4]   2> 68658 INFO  (zkCallback-73-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 68660 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 68660 INFO  (zkCallback-68-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 68660 INFO  (zkCallback-68-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 68660 INFO  (zkCallback-73-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 68668 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 68669 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 68670 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 68670 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/cores
   [junit4]   2> 68671 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 68671 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_D87C2A1E874D0DFE-001/shard-1-001/cores/collection1
   [junit4]   2> 68671 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 68673 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 68673 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[D87C2A1E874D0DFE]) [n:127.0.0.1:40003_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 68674 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 68674 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 68675 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.ZkController Collection collection1 not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 68675 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 68675 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40003",
   [junit4]   2>   "node_name":"127.0.0.1:40003_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 4
   [junit4]   2> 68675 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [collection1]
   [junit4]   2> 68675 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40003",
   [junit4]   2>   "node_name":"127.0.0.1:40003_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 68675 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 68676 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 68676 INFO  (OverseerStateUpdate-95812075095588868-127.0.0.1:53731_-n_0000000000) [n:127.0.0.1:53731_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 68778 INFO  (zkCallback-73-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 68778 INFO  (zkCallback-66-thread-1-processing-n:127.0.0.1:53731_) [n:127.0.0.1:53731_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 68778 INFO  (zkCallback-68-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 69676 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 69676 INFO  (coreLoadExecutor-303-thread-1-processing-n:127.0.0.1:40003_) [n:127.0.0.1:40003_ c:collection1   x:colle

[...truncated too long message...]

ess$200(LeaderElector.java:56)
   [junit4]   2> 	at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:348)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(SolrZkClient.java:266)
   [junit4]   2> 	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   [junit4]   2> 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 3930538 WARN  (zkCallback-3794-thread-2-processing-n:127.0.0.1:39387_solr) [n:127.0.0.1:39387_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3930536 WARN  (zkCallback-3817-thread-2-processing-n:127.0.0.1:42884_solr) [n:127.0.0.1:42884_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3930538 WARN  (zkCallback-3815-thread-2-processing-n:127.0.0.1:46941_solr) [n:127.0.0.1:46941_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 3930537 INFO  (jetty-launcher-3775-thread-13) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3359f584{/solr,null,UNAVAILABLE}
   [junit4]   2> 3930539 INFO  (jetty-launcher-3775-thread-7) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2883f82c{/solr,null,UNAVAILABLE}
   [junit4]   2> 3930539 INFO  (jetty-launcher-3775-thread-1) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@654d6eda{/solr,null,UNAVAILABLE}
   [junit4]   2> 3930539 INFO  (jetty-launcher-3775-thread-8) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@78bf24a3{/solr,null,UNAVAILABLE}
   [junit4]   2> 3930540 INFO  (jetty-launcher-3775-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3fb36723{/solr,null,UNAVAILABLE}
   [junit4]   2> 3930543 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[D87C2A1E874D0DFE]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34152 34152
   [junit4]   2> 3930721 INFO  (Thread-100393) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:34152 34152
   [junit4]   2> 3930722 WARN  (Thread-100393) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/aliases.json
   [junit4]   2> 	13	/solr/security.json
   [junit4]   2> 	11	/solr/configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 	11	/solr/configs/solrCloudCollectionConfig
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/clusterstate.json
   [junit4]   2> 	12	/solr/collections/test_col/state.json
   [junit4]   2> 	2	/solr/collections/test_col/leader_elect/shard2/election/95812325951733789-core_node11-n_0000000001
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	14	/solr/live_nodes
   [junit4]   2> 	14	/solr/collections
   [junit4]   2> 
   [junit4]   2> 3930722 INFO  (SUITE-TestTolerantUpdateProcessorRandomCloud-seed#[D87C2A1E874D0DFE]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Apr 30, 2016 1:35:08 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=HighCompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=20452, maxDocsPerChunk=3, blockSize=31), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=HIGH_COMPRESSION, chunkSize=20452, blockSize=31)), sim=ClassicSimilarity, locale=sk, timezone=Etc/GMT-7
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=199675744,total=515375104
   [junit4]   2> NOTE: All tests run in this JVM: [SolrCmdDistributorTest, SolrIndexSplitterTest, TestRequestStatusCollectionAPI, TimeZoneUtilsTest, SuggestComponentTest, DirectSolrSpellCheckerTest, TestTrie, CollectionsAPIDistributedZkTest, HdfsBasicDistributedZkTest, SoftAutoCommitTest, TestQueryUtils, AddSchemaFieldsUpdateProcessorFactoryTest, TestExclusionRuleCollectionAccess, TestCustomSort, DeleteReplicaTest, SpellingQueryConverterTest, TestAnalyzeInfixSuggestions, TestJsonFacets, CSVRequestHandlerTest, TriLevelCompositeIdRoutingTest, TestDynamicFieldCollectionResource, TestLFUCache, CloudMLTQParserTest, TolerantUpdateProcessorTest, TestSolrQueryResponse, TestUseDocValuesAsStored, TestSimpleTrackingShardHandler, TestObjectReleaseTracker, TestMacros, TestTestInjection, TestSolr4Spatial2, TestCoreContainer, TestPartialUpdateDeduplication, SecurityConfHandlerTest, TestReloadAndDeleteDocs, CollectionsAPIAsyncDistributedZkTest, DistributedExpandComponentTest, NumericFieldsTest, TestDFISimilarityFactory, TestLRUStatsCache, ShardRoutingCustomTest, SolrInfoMBeanTest, TestCodecSupport, TestSchemaVersionResource, TestSolrConfigHandlerCloud, ZkStateReaderTest, BasicFunctionalityTest, SolrIndexConfigTest, CdcrUpdateLogTest, SpellCheckCollatorTest, TestConfigOverlay, AliasIntegrationTest, TestMissingGroups, CustomCollectionTest, CollectionReloadTest, TestQuerySenderNoQuery, DocValuesTest, RecoveryAfterSoftCommitTest, UniqFieldsUpdateProcessorFactoryTest, FastVectorHighlighterTest, TestCloudInspectUtil, TestSolrDeletionPolicy2, LoggingHandlerTest, SaslZkACLProviderTest, TestAuthorizationFramework, HighlighterTest, DistribCursorPagingTest, FileUtilsTest, TestSortByMinMaxFunction, TestHashPartitioner, TestManagedSynonymFilterFactory, TestHashQParserPlugin, FacetPivotSmallTest, SchemaVersionSpecificBehaviorTest, ConvertedLegacyTest, TestDFRSimilarityFactory, BlockJoinFacetRandomTest, TestBinaryResponseWriter, DateRangeFieldTest, TestFreeTextSuggestions, HighlighterMaxOffsetTest, TestWordDelimiterFilterFactory, TestBlobHandler, TestSurroundQueryParser, NotRequiredUniqueKeyTest, OutOfBoxZkACLAndCredentialsProvidersTest, TestSchemaNameResource, HdfsThreadLeakTest, DocumentBuilderTest, TestGeoJSONResponseWriter, SpatialFilterTest, AnalyticsQueryTest, TermsComponentTest, HdfsCollectionsAPIDistributedZkTest, RuleEngineTest, MoreLikeThisHandlerTest, OverseerTest, RecoveryZkTest, ZkSolrClientTest, TestZkChroot, ZkCLITest, TestDistributedSearch, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, AsyncMigrateRouteKeyTest, CdcrReplicationHandlerTest, CollectionTooManyReplicasTest, ConcurrentDeleteAndCreateCollectionTest, DeleteStatusTest, DistributedQueueTest, HttpPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, RemoteQueryErrorTest, ShardSplitTest, TestOnReconnectListenerSupport, TestRebalanceLeaders, TestTolerantUpdateProcessorRandomCloud]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTolerantUpdateProcessorRandomCloud -Dtests.seed=D87C2A1E874D0DFE -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sk -Dtests.timezone=Etc/GMT-7 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestTolerantUpdateProcessorRandomCloud (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Could not find collection:test_col
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([D87C2A1E874D0DFE]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:150)
   [junit4]    > 	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorRandomCloud.java:135)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [474/592 (3!)] on J1 in 37.89s, 0 tests, 1 failure <<< FAILURES!

[...truncated 385 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 592 suites (10 ignored), 2397 tests, 1 suite-level error, 1 error, 1 failure, 31 ignored (16 assumptions) [seed: D87C2A1E874D0DFE]

Total time: 253 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 51 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/51/

3 tests failed.
FAILED:  org.apache.solr.cloud.RollingRestartTest.test

Error Message:
Unable to restart (#6): CloudJettyRunner [url=http://127.0.0.1:52780/lfu/collection1]

Stack Trace:
java.lang.AssertionError: Unable to restart (#6): CloudJettyRunner [url=http://127.0.0.1:52780/lfu/collection1]
	at __randomizedtesting.SeedInfo.seed([6C1927A853BBAADC:E44D1872FD47C724]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.RollingRestartTest.restartWithRolesTest(RollingRestartTest.java:103)
	at org.apache.solr.cloud.RollingRestartTest.test(RollingRestartTest.java:52)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestBlobHandler.doBlobHandlerTest

Error Message:
Could not find collection : .system

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : .system
	at __randomizedtesting.SeedInfo.seed([6C1927A853BBAADC:8CD805FAE857DC2E]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.handler.TestBlobHandler.doBlobHandlerTest(TestBlobHandler.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:44715/z

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:44715/z
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:416)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more




Build Log:
[...truncated 12465 lines...]
   [junit4] Suite: org.apache.solr.cloud.RollingRestartTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/init-core-data-001
   [junit4]   2> 3068400 INFO  (SUITE-RollingRestartTest-seed#[6C1927A853BBAADC]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 3068400 INFO  (SUITE-RollingRestartTest-seed#[6C1927A853BBAADC]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /lfu/
   [junit4]   2> 3068402 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3068402 INFO  (Thread-6969) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3068402 INFO  (Thread-6969) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3068502 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ZkTestServer start zk server on port:56566
   [junit4]   2> 3068502 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3068502 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3068504 INFO  (zkCallback-2327-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@355232dc name:ZooKeeperConnection Watcher:127.0.0.1:56566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3068504 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3068504 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3068504 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 3068506 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3068506 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3068507 INFO  (zkCallback-2328-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@66f5da7e name:ZooKeeperConnection Watcher:127.0.0.1:56566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3068507 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3068507 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3068507 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 3068508 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 3068508 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 3068509 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 3068510 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 3068510 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 3068511 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 3068511 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 3068512 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3068512 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 3068513 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 3068513 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 3068514 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 3068514 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 3068514 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 3068514 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 3068515 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 3068515 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 3068516 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3068516 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 3068517 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3068517 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 3068518 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 3068518 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 3068518 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 3068518 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 3068580 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1
   [junit4]   2> 3068581 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 3068582 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@bd491b4{/lfu,null,AVAILABLE}
   [junit4]   2> 3068584 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4eb8a854{HTTP/1.1,[http/1.1]}{127.0.0.1:39860}
   [junit4]   2> 3068584 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.Server Started @3070892ms
   [junit4]   2> 3068584 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/tempDir-001/control/data, hostContext=/lfu, hostPort=39860, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores}
   [junit4]   2> 3068584 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 3068585 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001'
   [junit4]   2> 3068585 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3068585 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3068585 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3068585 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3068586 INFO  (zkCallback-2329-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@46c52704 name:ZooKeeperConnection Watcher:127.0.0.1:56566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3068586 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3068586 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3068587 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3068587 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/solr.xml
   [junit4]   2> 3068590 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores
   [junit4]   2> 3068590 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CoreContainer New CoreContainer 149423379
   [junit4]   2> 3068590 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001]
   [junit4]   2> 3068590 WARN  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/lib
   [junit4]   2> 3068590 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 3068591 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3068591 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3068591 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3068591 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56566/solr
   [junit4]   2> 3068592 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3068592 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3068592 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3068593 INFO  (zkCallback-2332-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@23cab007 name:ZooKeeperConnection Watcher:127.0.0.1:56566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3068593 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3068593 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3068594 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$74/628170364@6b81ed33
   [junit4]   2> 3068594 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3068594 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4b693a6b name:ZooKeeperConnection Watcher:127.0.0.1:56566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3068595 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3068595 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 3068596 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 3068598 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 3068599 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 3068600 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 3068601 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 3068601 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 3068602 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 3068603 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 3068604 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3068604 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3068605 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 3068605 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 3068606 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3068606 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95804997858951172-127.0.0.1:39860_lfu-n_0000000000
   [junit4]   2> 3068607 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39860_lfu
   [junit4]   2> 3068607 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 3068607 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer Overseer (id=95804997858951172-127.0.0.1:39860_lfu-n_0000000000) starting
   [junit4]   2> 3068608 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 3068611 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 3068611 INFO  (OverseerCollectionConfigSetProcessor-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 3068611 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 3068611 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39860_lfu
   [junit4]   2> 3068612 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:39860_lfu
   [junit4]   2> 3068613 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 3068613 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 3068613 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3068625 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 3068625 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 3068625 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores
   [junit4]   2> 3068626 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 3068626 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1
   [junit4]   2> 3068626 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 3068627 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 3068627 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:39860_lfu    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 3068627 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 3068627 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 3068627 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 3068627 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 3068627 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:39860/lfu",
   [junit4]   2>   "node_name":"127.0.0.1:39860_lfu",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 3068628 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 3068628 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:39860/lfu",
   [junit4]   2>   "node_name":"127.0.0.1:39860_lfu",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3068628 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 3068628 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 3068628 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 3068629 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3069628 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 3069629 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 3069629 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 3069629 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 3069629 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 3069629 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1'
   [junit4]   2> 3069629 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3069630 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3069630 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 3069633 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 3069636 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 3069642 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 3069646 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 3069725 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 3069729 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 3069730 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 3069735 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 3069736 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 3069737 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3069737 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 3069738 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 3069738 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 3069738 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore solr.StandardDirectoryFactory
   [junit4]   2> 3069738 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/]
   [junit4]   2> 3069738 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@4aad92a
   [junit4]   2> 3069739 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data
   [junit4]   2> 3069739 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/index/
   [junit4]   2> 3069739 WARN  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 3069740 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/index
   [junit4]   2> 3069740 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@28158185
   [junit4]   2> 3069746 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 3069746 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 3069746 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3069748 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 3069749 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 3069749 WARN  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 3069750 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3069750 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3069751 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3069751 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 3069753 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 3069753 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 3069753 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 3069753 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 3069754 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 3069754 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 3069754 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=13, maxMergeAtOnceExplicit=35, maxMergedSegmentMB=1.8974609375, floorSegmentMB=0.494140625, forceMergeDeletesPctAllowed=26.138229044751252, segmentsPerTier=16.0, maxCFSSegmentSizeMB=0.2822265625, noCFSRatio=0.5015657907609108
   [junit4]   2> 3069755 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/control-001/cores/collection1/data/index,segFN=segments_1,generation=1}
   [junit4]   2> 3069755 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 3069755 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@450e999a[collection1] main]
   [junit4]   2> 3069755 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 3069755 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 3069756 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 3069757 INFO  (searcherExecutor-8520-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@450e999a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 3069757 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 3069757 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 3069757 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 3069757 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1532879967162990592
   [junit4]   2> 3069759 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1532879967162990592
   [junit4]   2> 3069759 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 3069759 INFO  (coreLoadExecutor-8519-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 3069759 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:39860/lfu collection:control_collection shard:shard1
   [junit4]   2> 3069759 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 3069761 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95804997858951172-core_node1-n_0000000000
   [junit4]   2> 3069761 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 3069762 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 3069762 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 3069762 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 3069762 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:39860/lfu/collection1/
   [junit4]   2> 3069762 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 3069762 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:39860/lfu/collection1/ has no replicas
   [junit4]   2> 3069763 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 3069764 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95804997858951172-core_node1-n_0000000000
   [junit4]   2> 3069765 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:39860/lfu/collection1/ shard1
   [junit4]   2> 3069765 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:39860/lfu",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 3069866 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3069915 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:39860/lfu/collection1/ and leader is http://127.0.0.1:39860/lfu/collection1/
   [junit4]   2> 3069916 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39860/lfu
   [junit4]   2> 3069916 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 3069916 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 3069916 INFO  (coreZkRegister-8513-thread-1-processing-n:127.0.0.1:39860_lfu x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:39860_lfu c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 3069917 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:39860/lfu",
   [junit4]   2>   "node_name":"127.0.0.1:39860_lfu",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 3069920 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:39860/lfu",
   [junit4]   2>   "node_name":"127.0.0.1:39860_lfu",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 3070022 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3070128 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:56566/solr
   [junit4]   2> 3070128 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3070128 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3070129 INFO  (zkCallback-2335-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@15bea059 name:ZooKeeperConnection Watcher:127.0.0.1:56566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3070129 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3070129 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3070129 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3070130 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3070130 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 3070186 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001/cores/collection1
   [junit4]   2> 3070186 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001
   [junit4]   2> 3070187 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 3070187 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7dbb44d9{/lfu,null,AVAILABLE}
   [junit4]   2> 3070188 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.ServerConnector Started ServerConnector@336ed624{HTTP/1.1,[http/1.1]}{127.0.0.1:48017}
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.e.j.s.Server Started @3072496ms
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/lfu, hostPort=48017, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001/cores}
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001'
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 3070189 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3070190 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3070191 INFO  (zkCallback-2336-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@435da401 name:ZooKeeperConnection Watcher:127.0.0.1:56566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3070191 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3070192 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3070192 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 3070192 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001/solr.xml
   [junit4]   2> 3070195 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001/cores
   [junit4]   2> 3070196 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CoreContainer New CoreContainer 873300638
   [junit4]   2> 3070196 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001]
   [junit4]   2> 3070196 WARN  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.RollingRestartTest_6C1927A853BBAADC-001/shard-1-001/lib
   [junit4]   2> 3070196 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:56566/solr
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 3070197 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3070198 INFO  (zkCallback-2339-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1fe29925 name:ZooKeeperConnection Watcher:127.0.0.1:56566 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3070198 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3070198 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 3070199 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$74/628170364@10648ae9
   [junit4]   2> 3070199 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3070200 INFO  (zkCallback-2340-thread-1-processing-n:127.0.0.1:48017_lfu) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@65e3aeb3 name:ZooKeeperConnection Watcher:127.0.0.1:56566/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 3070200 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3070203 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 3070203 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3070204 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 3070204 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95804997858951176-127.0.0.1:48017_lfu-n_0000000001
   [junit4]   2> 3070205 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95804997858951172-127.0.0.1:39860_lfu-n_0000000000 to know if I could be the leader
   [junit4]   2> 3070205 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.ZkController Publish node=127.0.0.1:48017_lfu as DOWN
   [junit4]   2> 3070206 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:48017_lfu"} current state version: 3
   [junit4]   2> 3070206 INFO  (OverseerStateUpdate-95804997858951172-127.0.0.1:39860_lfu-n_0000000000) [n:127.0.0.1:39860_lfu    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:48017_lfu
   [junit4]   2> 3070307 INFO  (zkCallback-2335-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3070307 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3070307 INFO  (zkCallback-2340-thread-1-processing-n:127.0.0.1:48017_lfu) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3071206 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:48017_lfu
   [junit4]   2> 3071206 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:48017_lfu
   [junit4]   2> 3071207 INFO  (zkCallback-2335-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3071207 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3071207 INFO  (zkCallback-2340-thread-1-processing-n:127.0.0.1:48017_lfu) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 3071207 INFO  (zkCallback-2333-thread-1-processing-n:127.0.0.1:39860_lfu) [n:127.0.0.1:39860_lfu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3071207 INFO  (zkCallback-2340-thread-1-processing-n:127.0.0.1:48017_lfu) [n:127.0.0.1:48017_lfu    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3071211 INFO  (zkCallback-2335-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3071224 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 3071224 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.1:48017_lfu    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 3071225 INFO  (TEST-RollingRestartTest.test-seed#[6C1927A853BBAADC]) [n:127.0.0.

[...truncated too long message...]

pl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 5552732 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[6C1927A853BBAADC]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_6C1927A853BBAADC-001
   [junit4]   2> Apr 28, 2016 7:12:17 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60), sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=is-IS, timezone=Australia/Melbourne
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=147999152,total=510132224
   [junit4]   2> NOTE: All tests run in this JVM: [TestRestoreCore, SpatialHeatmapFacetsTest, TestUseDocValuesAsStored2, BasicAuthIntegrationTest, OpenExchangeRatesOrgProviderTest, TestSolrCloudWithKerberosAlt, ResponseHeaderTest, BasicFunctionalityTest, TestBlendedInfixSuggestions, ConvertedLegacyTest, ZkSolrClientTest, HdfsNNFailoverTest, UpdateRequestProcessorFactoryTest, SegmentsInfoRequestHandlerTest, HdfsTlogReplayBufferedWhileIndexingTest, SortSpecParsingTest, UniqFieldsUpdateProcessorFactoryTest, TestCollapseQParserPlugin, TestDocumentBuilder, TestStressVersions, IndexBasedSpellCheckerTest, NotRequiredUniqueKeyTest, AddBlockUpdateTest, SpellCheckCollatorTest, LeaderInitiatedRecoveryOnShardRestartTest, CursorMarkTest, DistributedIntervalFacetingTest, SchemaVersionSpecificBehaviorTest, TestRandomCollapseQParserPlugin, TestRequestStatusCollectionAPI, TestTrackingShardHandlerFactory, TestSolrFieldCacheMBean, TestMiniSolrCloudCluster, TestSolrXml, TermsComponentTest, TestSystemIdResolver, SpellCheckComponentTest, TestExclusionRuleCollectionAccess, MBeansHandlerTest, TestQueryTypes, TestSurroundQueryParser, AsyncMigrateRouteKeyTest, TestFieldSortValues, ZkStateWriterTest, SpatialFilterTest, HighlighterTest, OverseerRolesTest, PreAnalyzedUpdateProcessorTest, TestShardHandlerFactory, SolrCmdDistributorTest, TestSolrDeletionPolicy2, TestQueryUtils, TestScoreJoinQPScore, BadCopyFieldTest, TestDynamicLoading, DocValuesMissingTest, ChaosMonkeySafeLeaderTest, ClusterStateUpdateTest, BJQParserTest, MultiTermTest, WordBreakSolrSpellCheckerTest, MoreLikeThisHandlerTest, TestCursorMarkWithoutUniqueKey, StressHdfsTest, TestFiltering, TestMinMaxOnMultiValuedField, SolrInfoMBeanTest, RecoveryAfterSoftCommitTest, FastVectorHighlighterTest, SimplePostToolTest, TestManagedSchemaAPI, ZkControllerTest, MinimalSchemaTest, TestSchemaResource, BigEndianAscendingWordDeserializerTest, DistribCursorPagingTest, HttpPartitionTest, TestRawResponseWriter, TestEmbeddedSolrServerConstructors, TestFastLRUCache, TestUniqueKeyFieldResource, TestSolrCoreProperties, FileUtilsTest, DocumentBuilderTest, SmileWriterTest, ShardRoutingTest, SolrIndexSplitterTest, DistributedQueryComponentOptimizationTest, SaslZkACLProviderTest, ConfigSetsAPITest, CollectionsAPIDistributedZkTest, TestSortingResponseWriter, ExplicitHLLTest, LoggingHandlerTest, TestQuerySenderListener, TestLFUCache, SpellPossibilityIteratorTest, DistributedFacetPivotLargeTest, HLLUtilTest, TestDistributedGrouping, TestHashPartitioner, TermVectorComponentDistributedTest, TestStressReorder, TestJoin, HardAutoCommitTest, DistributedTermsComponentTest, TestCoreContainer, SimpleFacetsTest, TestSolr4Spatial, PeerSyncTest, TestLazyCores, TestIndexSearcher, CurrencyFieldOpenExchangeTest, DistributedQueryElevationComponentTest, TestCoreDiscovery, SuggesterFSTTest, DocValuesTest, TermVectorComponentTest, RegexBoostProcessorTest, TestJmxIntegration, JSONWriterTest, TestPartialUpdateDeduplication, PingRequestHandlerTest, SOLR749Test, TestSolrIndexConfig, SolrIndexConfigTest, NumericFieldsTest, TestFuzzyAnalyzedSuggestions, ExternalFileFieldSortTest, TestCodecSupport, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TimeZoneUtilsTest, ResourceLoaderTest, PluginInfoTest, PreAnalyzedFieldTest, PrimUtilsTest, SpellingQueryConverterTest, DOMUtilTest, RAMDirectoryFactoryTest, ClusterStateTest, TestUtils, ZkNodePropsTest, SliceStateTest, DistributedMLTComponentTest, TestRTGBase, TestSimpleTrackingShardHandler, ConnectionReuseTest, CdcrReplicationHandlerTest, CloudExitableDirectoryReaderTest, CollectionReloadTest, CollectionsAPIAsyncDistributedZkTest, ConnectionManagerTest, CustomCollectionTest, DeleteShardTest, DistributedQueueTest, ForceLeaderTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, ReplicationFactorTest, RestartWhileUpdatingTest, SharedFSAutoReplicaFailoverTest, SharedFSAutoReplicaFailoverUtilsTest, SolrXmlInZkTest, TestCloudDeleteByQuery, TestCollectionAPI, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestMiniSolrCloudClusterBase, TestRebalanceLeaders, TestRequestForwarding, TestStressLiveNodes, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZkTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [591/591 (3!)] on J1 in 1715.09s, 1 test, 1 error <<< FAILURES!

[...truncated 1 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J1-20160428_173942_717.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid25977.hprof ...
   [junit4] Heap dump file created [715827357 bytes in 11.471 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 12 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 591 suites (10 ignored), 2393 tests, 2 errors, 1 failure, 31 ignored (16 assumptions) [seed: 6C1927A853BBAADC]

Total time: 222 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 50 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/50/

3 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=20956, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=20956, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45363/xhzm: collection already exists: awholynewstresscollection_collection1_0
	at __randomizedtesting.SeedInfo.seed([A06829637BC1E211]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:987)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=123387, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=123387, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:38921/_yrh: collection already exists: awholynewstresscollection_collection2_1
	at __randomizedtesting.SeedInfo.seed([A06829637BC1E211]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:987)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 11 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 11 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([A06829637BC1E211]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
	at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11623 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/init-core-data-001
   [junit4]   2> 1249848 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[A06829637BC1E211]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 1249848 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[A06829637BC1E211]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /xhzm/
   [junit4]   2> 1249852 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1249852 INFO  (Thread-13251) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1249852 INFO  (Thread-13251) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1249952 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ZkTestServer start zk server on port:49831
   [junit4]   2> 1249953 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1249954 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1249955 INFO  (zkCallback-1558-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7e9ddee8 name:ZooKeeperConnection Watcher:127.0.0.1:49831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1249955 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1249955 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1249955 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1249957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1249957 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1249958 INFO  (zkCallback-1559-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@a1af0d1 name:ZooKeeperConnection Watcher:127.0.0.1:49831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1249958 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1249958 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1249958 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1249960 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1249961 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1249962 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1249963 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1249963 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1249965 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1249965 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1249966 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1249966 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1249967 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1249967 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1249968 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1249968 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1249969 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1249970 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1249971 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1249971 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1249972 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1249972 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1249973 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1249973 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1249974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1249974 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1249976 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1249976 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1250039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1
   [junit4]   2> 1250040 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1250041 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@79c3d35a{/xhzm,null,AVAILABLE}
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3cf3e23d{HTTP/1.1,[http/1.1]}{127.0.0.1:?????}
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.Server Started @???????ms
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/tempDir-001/control/data, hostContext=/xhzm, hostPort=45805, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores}
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001'
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1250050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1250051 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250051 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250051 INFO  (zkCallback-1560-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7487ba32 name:ZooKeeperConnection Watcher:127.0.0.1:49831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250052 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250052 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250052 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1250052 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/solr.xml
   [junit4]   2> 1250056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores
   [junit4]   2> 1250056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CoreContainer New CoreContainer 1806641505
   [junit4]   2> 1250056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001]
   [junit4]   2> 1250056 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/lib
   [junit4]   2> 1250056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49831/solr
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1250057 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250058 INFO  (zkCallback-1563-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@63d6b10a name:ZooKeeperConnection Watcher:127.0.0.1:49831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250058 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250059 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1250061 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/1561110113@b6ca318
   [junit4]   2> 1250061 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1250062 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@58665241 name:ZooKeeperConnection Watcher:127.0.0.1:49831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1250062 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1250063 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1250064 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1250066 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1250067 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1250068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1250071 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1250072 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1250072 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1250073 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1250073 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1250074 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1250076 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1250077 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1250077 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1250077 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95799072864075780-127.0.0.1:45805_xhzm-n_0000000000
   [junit4]   2> 1250078 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45805_xhzm
   [junit4]   2> 1250078 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1250078 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer Overseer (id=95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) starting
   [junit4]   2> 1250079 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1250082 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1250082 INFO  (OverseerCollectionConfigSetProcessor-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1250082 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1250082 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45805_xhzm
   [junit4]   2> 1250090 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:45805_xhzm
   [junit4]   2> 1250091 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1250091 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1250094 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1250100 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1250100 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1250100 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores
   [junit4]   2> 1250101 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1250101 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1
   [junit4]   2> 1250101 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1250102 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1250102 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45805_xhzm    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1250102 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1250102 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1250102 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1250102 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 1250103 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 1250103 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45805/xhzm",
   [junit4]   2>   "node_name":"127.0.0.1:45805_xhzm",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1250103 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1250103 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45805/xhzm",
   [junit4]   2>   "node_name":"127.0.0.1:45805_xhzm",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1250104 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1250104 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1250105 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1251103 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1'
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1251104 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1251105 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1251108 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1251111 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1251118 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1251123 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1251212 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1251216 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1251217 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1251222 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1251223 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1251225 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1251225 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1251226 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1251226 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1251226 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1251226 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data/]
   [junit4]   2> 1251226 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ec93a98
   [junit4]   2> 1251227 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data
   [junit4]   2> 1251227 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1251227 WARN  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1251227 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data/index
   [junit4]   2> 1251227 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=29, maxMergedSegmentMB=1.8359375, floorSegmentMB=1.4501953125, forceMergeDeletesPctAllowed=20.96224958701808, segmentsPerTier=29.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.6607549447692009
   [junit4]   2> 1251228 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@61015113 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8d3a315),segFN=segments_1,generation=1}
   [junit4]   2> 1251228 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1251228 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data/
   [junit4]   2> 1251228 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1251230 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1251230 WARN  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1251232 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1251232 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1251233 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1251233 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1251235 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1251235 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1251235 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1251235 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1251236 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1251236 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1251236 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.2294921875, noCFSRatio=0.42439390548904665]
   [junit4]   2> 1251236 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(RAMDirectory@61015113 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@8d3a315),segFN=segments_1,generation=1}
   [junit4]   2> 1251236 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1251236 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6530b99c[collection1] main]
   [junit4]   2> 1251237 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1251239 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1251240 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1251240 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1251240 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1251240 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1251240 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1532785167279587328
   [junit4]   2> 1251241 INFO  (searcherExecutor-4338-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6530b99c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1251242 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 2.0ms to seed version buckets with highest version 1532785167279587328
   [junit4]   2> 1251242 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1251242 INFO  (coreLoadExecutor-4337-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1251243 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:45805/xhzm collection:control_collection shard:shard1
   [junit4]   2> 1251243 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1251245 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95799072864075780-core_node1-n_0000000000
   [junit4]   2> 1251246 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1251247 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1251247 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1251247 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:45805/xhzm/collection1/
   [junit4]   2> 1251247 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1251247 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1251247 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:45805/xhzm/collection1/ has no replicas
   [junit4]   2> 1251247 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1251248 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95799072864075780-core_node1-n_0000000000
   [junit4]   2> 1251249 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:45805/xhzm/collection1/ shard1
   [junit4]   2> 1251250 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:45805/xhzm",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1251351 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1251400 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:45805/xhzm/collection1/ and leader is http://127.0.0.1:45805/xhzm/collection1/
   [junit4]   2> 1251400 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45805/xhzm
   [junit4]   2> 1251400 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1251400 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1251400 INFO  (coreZkRegister-4331-thread-1-processing-n:127.0.0.1:45805_xhzm x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:45805_xhzm c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1251402 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45805/xhzm",
   [junit4]   2>   "node_name":"127.0.0.1:45805_xhzm",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1251402 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45805/xhzm",
   [junit4]   2>   "node_name":"127.0.0.1:45805_xhzm",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1251504 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1251603 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:49831/solr
   [junit4]   2> 1251603 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1251603 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1251604 INFO  (zkCallback-1566-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@262b3cf2 name:ZooKeeperConnection Watcher:127.0.0.1:49831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1251604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1251604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1251604 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1251605 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1251606 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1251679 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/cores/collection1
   [junit4]   2> 1251680 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001
   [junit4]   2> 1251680 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1251687 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6b6a640a{/xhzm,null,AVAILABLE}
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.ServerConnector Started ServerConnector@1b0b6f7b{HTTP/1.1,[http/1.1]}{127.0.0.1:?????}
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.e.j.s.Server Started @???????ms
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/xhzm, hostPort=45363, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/cores}
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001'
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1251688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1251693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1251694 INFO  (zkCallback-1567-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@401fe75d name:ZooKeeperConnection Watcher:127.0.0.1:49831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1251695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1251695 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1251703 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1251703 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/solr.xml
   [junit4]   2> 1251706 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/cores
   [junit4]   2> 1251706 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CoreContainer New CoreContainer 184594453
   [junit4]   2> 1251707 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001]
   [junit4]   2> 1251707 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/lib
   [junit4]   2> 1251707 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1251708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1251708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1251708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1251708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:49831/solr
   [junit4]   2> 1251708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1251708 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1251709 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1251709 INFO  (zkCallback-1570-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@29d075a9 name:ZooKeeperConnection Watcher:127.0.0.1:49831 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1251710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1251710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1251710 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$10/1561110113@4594136c
   [junit4]   2> 1251711 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1251712 INFO  (zkCallback-1571-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@74560db6 name:ZooKeeperConnection Watcher:127.0.0.1:49831/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1251712 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1251717 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1251718 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1251720 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1251720 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95799072864075784-127.0.0.1:45363_xhzm-n_0000000001
   [junit4]   2> 1251721 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95799072864075780-127.0.0.1:45805_xhzm-n_0000000000 to know if I could be the leader
   [junit4]   2> 1251721 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.ZkController Publish node=127.0.0.1:45363_xhzm as DOWN
   [junit4]   2> 1251722 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:45363_xhzm"} current state version: 3
   [junit4]   2> 1251722 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:45363_xhzm
   [junit4]   2> 1251824 INFO  (zkCallback-1566-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1251824 INFO  (zkCallback-1571-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1251824 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252723 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45363_xhzm
   [junit4]   2> 1252723 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:45363_xhzm
   [junit4]   2> 1252724 INFO  (zkCallback-1566-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252724 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252724 INFO  (zkCallback-1571-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1252724 INFO  (zkCallback-1566-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1252724 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1252724 INFO  (zkCallback-1571-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1252732 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1252732 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1252733 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/cores
   [junit4]   2> 1252735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1252735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E211-001/shard-1-001/cores/collection1
   [junit4]   2> 1252735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1252745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1252745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A06829637BC1E211]) [n:127.0.0.1:45363_xhzm    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1252749 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1252749 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1252749 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController Collection collection1 not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1252749 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 1252749 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45363/xhzm",
   [junit4]   2>   "node_name":"127.0.0.1:45363_xhzm",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 4
   [junit4]   2> 1252750 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:45363/xhzm",
   [junit4]   2>   "node_name":"127.0.0.1:45363_xhzm",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1252750 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1252750 INFO  (OverseerStateUpdate-95799072864075780-127.0.0.1:45805_xhzm-n_0000000000) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1252750 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [collection1]
   [junit4]   2> 1252750 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1252851 INFO  (zkCallback-1566-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1252851 INFO  (zkCallback-1564-thread-1-processing-n:127.0.0.1:45805_xhzm) [n:127.0.0.1:45805_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1252851 INFO  (zkCallback-1571-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1253751 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1253751 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController Check for collection zkNode:collection1
   [junit4]   2> 1253751 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1253751 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/collection1]
   [junit4]   2> 1253752 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/collection1] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1253752 INFO  (coreLoadExecutor-4348-thread-1-processing-n:127.0.0.1:45363_xhzm) [n:127.0.0.1:45363_xhzm c:collection1   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A06829637BC1E2

[...truncated too long message...]

wnClass(HdfsTestUtil.java:198)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.teardownClass(HdfsCollectionsAPIDistributedZkTest.java:52)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 4153968 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[A06829637BC1E211]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Apr 27, 2016 6:12:29 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 3 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=FSTOrd50, range_facet_l_dv=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), _version_=FSTOrd50, multiDefault=Lucene50(blocksize=128), a_t=Lucene50(blocksize=128), intDefault=FSTOrd50, a_i1=Lucene50(blocksize=128), id=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128))), range_facet_i_dv=Lucene50(blocksize=128), text=PostingsFormat(name=MockRandom), range_facet_l=Lucene50(blocksize=128), timestamp=Lucene50(blocksize=128)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Memory), timestamp=DocValuesFormat(name=Memory)}, maxPointsInLeafNode=549, maxMBSortInHeap=6.8592995711028895, sim=ClassicSimilarity, locale=el-CY, timezone=America/Nipigon
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=71011048,total=471334912
   [junit4]   2> NOTE: All tests run in this JVM: [TestExactStatsCache, TestManagedResource, HdfsChaosMonkeySafeLeaderTest, TestOnReconnectListenerSupport, BaseCdcrDistributedZkTest, TestJmxMonitoredMap, TestNRTOpen, CleanupOldIndexTest, TestConfigSetsAPIExclusivity, DirectUpdateHandlerTest, TestReloadDeadlock, SampleTest, TestReversedWildcardFilterFactory, TestAuthenticationFramework, TermVectorComponentDistributedTest, TestDocBasedVersionConstraints, TestMiniSolrCloudClusterBase, StatsComponentTest, BlobRepositoryCloudTest, HdfsBasicDistributedZk2Test, ConnectionManagerTest, TestDefaultStatsCache, TestConfigSetProperties, TestRuleBasedAuthorizationPlugin, TestSweetSpotSimilarityFactory, TestRecovery, TestStressLiveNodes, TestPerFieldSimilarityClassic, SolrCoreCheckLockOnStartupTest, TestSort, BigEndianAscendingWordSerializerTest, TestClusterStateMutator, RequestLoggingTest, CoreAdminCreateDiscoverTest, SimpleCollectionCreateDeleteTest, BlockJoinFacetDistribTest, SearchHandlerTest, TestSearchPerf, HardAutoCommitTest, ClusterStateTest, ShardSplitTest, CdcrRequestHandlerTest, AnalyticsMergeStrategyTest, TestFileDictionaryLookup, PKIAuthenticationIntegrationTest, OverseerStatusTest, TestSolrJ, TestZkChroot, TestSolrQueryParserDefaultOperatorResource, DeleteInactiveReplicaTest, DistributedFacetPivotWhiteBoxTest, CurrencyFieldXmlFileTest, TestStressRecovery, DeleteShardTest, XmlUpdateRequestHandlerTest, TestCloudManagedSchema, TestExtendedDismaxParser, TestComplexPhraseQParserPlugin, TestSuggestSpellingConverter, TestMaxScoreQueryParser, TestIndexingPerformance, HdfsRecoverLeaseTest, TestReload, TestExpandComponent, IgnoreCommitOptimizeUpdateProcessorFactoryTest, GraphQueryTest, DistributedSuggestComponentTest, LeaderElectionTest, QueryParsingTest, TestPostingsSolrHighlighter, TestJmxIntegration, TestOrdValues, TestCSVResponseWriter, ShowFileRequestHandlerTest, TestSolrQueryParser, TestIntervalFaceting, TestSolrDeletionPolicy1, TestShortCircuitedRequests, RollingRestartTest, TestSmileRequest, TestXmlQParser, HighlighterConfigTest, StatelessScriptUpdateProcessorFactoryTest, HdfsDirectoryTest, AnalysisErrorHandlingTest, TestDefaultSearchFieldResource, TestImplicitCoreProperties, TestBulkSchemaAPI, SuggestComponentContextFilterQueryTest, BasicDistributedZk2Test, BinaryUpdateRequestHandlerTest, TestGroupingSearch, TestWriterPerf, OverriddenZkACLAndCredentialsProvidersTest, JavabinLoaderTest, SimpleFacetsTest, RegexBoostProcessorTest, UnloadDistributedZkTest, SyncSliceTest, OverseerTest, LeaderElectionIntegrationTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, ZkSolrClientTest, ZkCLITest, ShardRoutingCustomTest, AutoCommitTest, TestStressReorder, TestJoin, TestCoreContainer, TestSolr4Spatial, PeerSyncTest, BadIndexSchemaTest, ConvertedLegacyTest, TestFiltering, TestIndexSearcher, HighlighterTest, SoftAutoCommitTest, DocValuesTest, SortByFunctionTest, DocValuesMultiTest, SolrInfoMBeanTest, DebugComponentTest, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, TestQueryTypes, TestOmitPositions, RequestHandlersTest, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, SolrPluginUtilsTest, MBeansHandlerTest, TestPHPSerializedResponseWriter, TestBinaryResponseWriter, UpdateParamsTest, TestSolrIndexConfig, NumericFieldsTest, TestElisionMultitermQuery, OutputWriterTest, ExternalFileFieldSortTest, NotRequiredUniqueKeyTest, SynonymTokenizerTest, TestDFRSimilarityFactory, TestLMDirichletSimilarityFactory, TestBM25SimilarityFactory, TimeZoneUtilsTest, TestFastOutputStream, OpenExchangeRatesOrgProviderTest, URLClassifyProcessorTest, PluginInfoTest, TestFastLRUCache, TestSystemIdResolver, PrimUtilsTest, DOMUtilTest, TestUtils, SliceStateTest, UUIDFieldTest, CircularListTest, TestRTGBase, DistributedIntervalFacetingTest, CdcrReplicationHandlerTest, DistribDocExpirationUpdateProcessorTest, ForceLeaderTest, LeaderInitiatedRecoveryOnShardRestartTest, ReplicationFactorTest, RestartWhileUpdatingTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [481/591 (3!)] on J2 in 583.49s, 1 test, 1 error <<< FAILURES!

[...truncated 349 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 591 suites (10 ignored), 2382 tests, 1 suite-level error, 2 errors, 31 ignored (16 assumptions) [seed: A06829637BC1E211]

Total time: 228 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 49 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/49/

1 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
	at __randomizedtesting.SeedInfo.seed([29242E7382F60DE5]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
	at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11462 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/init-core-data-001
   [junit4]   2> 1622341 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1622341 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1622425 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1622464 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1622480 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1622503 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_54521_hdfs____5bi6zj/webapp
   [junit4]   2> 1622669 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1623164 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54521
   [junit4]   2> 1623679 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1623680 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1623711 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_56145_datanode____n49cwo/webapp
   [junit4]   2> 1624292 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1624637 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56145
   [junit4]   2> 1624755 WARN  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1624759 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1624777 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47744_datanode____4odnjv/webapp
   [junit4]   2> 1624902 INFO  (IPC Server handler 4 on 49687) [    ] BlockStateChange BLOCK* processReport: from storage DS-e3bc9d2d-969a-4203-af71-a1842fc40a48 node DatanodeRegistration(127.0.0.1, datanodeUuid=88ce246c-19d0-40b0-9a16-ec36695954ce, infoPort=56145, ipcPort=38013, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1624902 INFO  (IPC Server handler 4 on 49687) [    ] BlockStateChange BLOCK* processReport: from storage DS-fdb9ae2f-1257-415a-be25-5e0f2d9f1f64 node DatanodeRegistration(127.0.0.1, datanodeUuid=88ce246c-19d0-40b0-9a16-ec36695954ce, infoPort=56145, ipcPort=38013, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1625055 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1625800 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47744
   [junit4]   2> 1626152 INFO  (IPC Server handler 5 on 49687) [    ] BlockStateChange BLOCK* processReport: from storage DS-28498c57-0104-4d22-96ea-47b607dcd51b node DatanodeRegistration(127.0.0.1, datanodeUuid=a650ff14-ef3c-416d-96f6-3d03c79cf86f, infoPort=47744, ipcPort=59950, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 1626152 INFO  (IPC Server handler 5 on 49687) [    ] BlockStateChange BLOCK* processReport: from storage DS-8d49033e-ebb6-4f06-b44b-83f6619be14c node DatanodeRegistration(127.0.0.1, datanodeUuid=a650ff14-ef3c-416d-96f6-3d03c79cf86f, infoPort=47744, ipcPort=59950, storageInfo=lv=-56;cid=testClusterID;nsid=257023781;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1626290 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1626298 INFO  (Thread-5745) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1626298 INFO  (Thread-5745) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1626398 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ZkTestServer start zk server on port:42830
   [junit4]   2> 1626398 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1626417 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1626429 INFO  (zkCallback-966-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1db9f962 name:ZooKeeperConnection Watcher:127.0.0.1:42830 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1626429 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1626430 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1626430 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1626437 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1626437 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1626495 INFO  (zkCallback-967-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@126f1b46 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1626495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1626495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1626495 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1626496 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1626497 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1626497 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1626504 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1626505 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1626506 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1626506 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1626507 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1626507 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1626510 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1626510 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1626511 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1626511 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1626511 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1626511 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1626512 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1626512 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1626513 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1626513 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1626513 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1626514 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1626520 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1626520 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1626521 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1626521 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1629060 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1
   [junit4]   2> 1629062 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1629064 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@724bbdc4{/,null,AVAILABLE}
   [junit4]   2> 1629073 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.ServerConnector Started ServerConnector@a5af8bb{HTTP/1.1,[http/1.1]}{127.0.0.1:54816}
   [junit4]   2> 1629073 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.Server Started @1631369ms
   [junit4]   2> 1629073 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49687/hdfs__localhost_49687__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001_tempDir-002_control_data, hostContext=/, hostPort=54816, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores}
   [junit4]   2> 1629073 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1629073 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001'
   [junit4]   2> 1629074 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1629074 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1629074 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1629080 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1629097 INFO  (zkCallback-968-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7a502592 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1629097 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1629098 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1629099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1629099 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/solr.xml
   [junit4]   2> 1629102 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores
   [junit4]   2> 1629102 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CoreContainer New CoreContainer 1498200714
   [junit4]   2> 1629103 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001]
   [junit4]   2> 1629103 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/lib
   [junit4]   2> 1629103 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1629107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1629107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1629107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1629107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42830/solr
   [junit4]   2> 1629107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1629107 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1629110 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1629114 INFO  (zkCallback-971-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@41aa8fa name:ZooKeeperConnection Watcher:127.0.0.1:42830 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1629115 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1629115 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1629120 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1163214024@4b5587e7
   [junit4]   2> 1629125 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1629137 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6ba235b0 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1629137 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1629138 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1629139 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1629141 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1629142 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1629143 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1629143 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1629144 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1629145 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1629145 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1629146 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1629146 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1629147 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1629148 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1629149 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1629149 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95793786670874628-127.0.0.1:54816_-n_0000000000
   [junit4]   2> 1629150 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54816_
   [junit4]   2> 1629150 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1629150 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer Overseer (id=95793786670874628-127.0.0.1:54816_-n_0000000000) starting
   [junit4]   2> 1629152 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1629172 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1629180 INFO  (OverseerCollectionConfigSetProcessor-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1629180 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1629181 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1629182 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54816_
   [junit4]   2> 1629183 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:54816_
   [junit4]   2> 1629184 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1629185 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1629198 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1629199 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1629199 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores
   [junit4]   2> 1629212 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1629212 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1
   [junit4]   2> 1629212 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1629222 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1629222 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1629222 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1629222 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 1629223 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 1629223 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1629223 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54816",
   [junit4]   2>   "node_name":"127.0.0.1:54816_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1629224 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54816",
   [junit4]   2>   "node_name":"127.0.0.1:54816_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1629224 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1629224 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1629225 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1629226 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1629226 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1629226 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1629226 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1629226 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1629226 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1'
   [junit4]   2> 1629227 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1629227 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1629228 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1629241 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 1629241 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:54816_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1629242 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1629254 WARN  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1629254 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1629261 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1629264 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1629358 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1629359 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1629359 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1629361 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1629389 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 1629389 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1629389 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 1629389 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49687/solr_hdfs_home
   [junit4]   2> 1629389 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1629390 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/control-001/cores/collection1], dataDir=[hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1629390 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@180baf58
   [junit4]   2> 1629390 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1629418 WARN  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 1634535 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1634536 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 1634552 WARN  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1634552 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1634561 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1634562 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 1634574 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1634574 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1634580 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=77.9541015625, floorSegmentMB=1.2529296875, forceMergeDeletesPctAllowed=22.82809990582344, segmentsPerTier=23.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.786929286943679
   [junit4]   2> 1634668 INFO  (IPC Server handler 2 on 49687) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59582 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3bc9d2d-969a-4203-af71-a1842fc40a48:NORMAL:127.0.0.1:32787|RBW], ReplicaUnderConstruction[[DISK]DS-28498c57-0104-4d22-96ea-47b607dcd51b:NORMAL:127.0.0.1:59582|FINALIZED]]} size 0
   [junit4]   2> 1634672 INFO  (IPC Server handler 1 on 49687) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:32787 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3bc9d2d-969a-4203-af71-a1842fc40a48:NORMAL:127.0.0.1:32787|RBW], ReplicaUnderConstruction[[DISK]DS-28498c57-0104-4d22-96ea-47b607dcd51b:NORMAL:127.0.0.1:59582|FINALIZED]]} size 0
   [junit4]   2> 1634674 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66dee619); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1634674 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1634688 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1634690 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1634691 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1634691 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1634691 WARN  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1634692 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1634692 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1634693 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1634693 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1634695 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1634695 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1634699 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1634699 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1634699 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1634715 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 15000ms; 
   [junit4]   2> 1634715 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1634720 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=33, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=50.3974609375, floorSegmentMB=1.275390625, forceMergeDeletesPctAllowed=5.693172197839953, segmentsPerTier=39.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0
   [junit4]   2> 1634738 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@66dee619); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1634738 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1634738 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@9712526[collection1] main]
   [junit4]   2> 1634738 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1634739 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1634740 INFO  (searcherExecutor-2239-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@9712526[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1634740 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1634740 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1634740 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1634740 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1532700595575586816
   [junit4]   2> 1634742 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1532700595575586816
   [junit4]   2> 1634742 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1634743 INFO  (coreLoadExecutor-2238-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1634756 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:54816 collection:control_collection shard:shard1
   [junit4]   2> 1634756 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1634757 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95793786670874628-core_node1-n_0000000000
   [junit4]   2> 1634758 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 1634763 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1634763 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1634763 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54816/collection1/
   [junit4]   2> 1634763 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1634763 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:54816/collection1/ has no replicas
   [junit4]   2> 1634763 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1634763 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1634763 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1634765 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95793786670874628-core_node1-n_0000000000
   [junit4]   2> 1634766 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54816/collection1/ shard1
   [junit4]   2> 1634766 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:54816",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1634867 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1634916 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:54816/collection1/ and leader is http://127.0.0.1:54816/collection1/
   [junit4]   2> 1634916 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54816
   [junit4]   2> 1634917 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1634917 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1634917 INFO  (coreZkRegister-2232-thread-1-processing-n:127.0.0.1:54816_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54816_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1634919 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "dataDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54816",
   [junit4]   2>   "node_name":"127.0.0.1:54816_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "ulogDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/tlog"} current state version: 3
   [junit4]   2> 1634919 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "dataDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54816",
   [junit4]   2>   "node_name":"127.0.0.1:54816_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state",
   [junit4]   2>   "ulogDir":"hdfs://localhost:49687/solr_hdfs_home/control_collection/core_node1/data/tlog"}
   [junit4]   2> 1635021 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1635243 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:42830/solr
   [junit4]   2> 1635243 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1635249 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1635260 INFO  (zkCallback-974-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19a5442d name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1635260 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1635260 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1635260 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1635261 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1635261 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1635404 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores/collection1
   [junit4]   2> 1635404 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001
   [junit4]   2> 1635405 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1635449 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@63be0ebd{/,null,AVAILABLE}
   [junit4]   2> 1635449 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.ServerConnector Started ServerConnector@5fdc1b04{HTTP/1.1,[http/1.1]}{127.0.0.1:42574}
   [junit4]   2> 1635449 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.e.j.s.Server Started @1637745ms
   [junit4]   2> 1635450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:49687/hdfs__localhost_49687__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J2_temp_solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=42574, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores}
   [junit4]   2> 1635450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1635450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001'
   [junit4]   2> 1635450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1635450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1635450 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1635466 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1635487 INFO  (zkCallback-975-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2a1bba4 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1635487 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1635487 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1635494 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1635494 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/solr.xml
   [junit4]   2> 1635498 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores
   [junit4]   2> 1635498 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CoreContainer New CoreContainer 788676558
   [junit4]   2> 1635498 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001]
   [junit4]   2> 1635498 WARN  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/lib
   [junit4]   2> 1635499 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1635500 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1635501 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1635501 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1635501 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42830/solr
   [junit4]   2> 1635501 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1635501 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1635503 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1635524 INFO  (zkCallback-978-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2154d36b name:ZooKeeperConnection Watcher:127.0.0.1:42830 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1635683 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1635684 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1635686 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$45/1163214024@595696bf
   [junit4]   2> 1635699 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1635718 INFO  (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2fea5e95 name:ZooKeeperConnection Watcher:127.0.0.1:42830/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1635719 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1635727 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1635728 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1635730 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1635732 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95793786670874632-127.0.0.1:42574_-n_0000000001
   [junit4]   2> 1635736 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95793786670874628-127.0.0.1:54816_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1635737 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.ZkController Publish node=127.0.0.1:42574_ as DOWN
   [junit4]   2> 1635738 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:42574_"} current state version: 4
   [junit4]   2> 1635738 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:42574_
   [junit4]   2> 1635840 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1635840 INFO  (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1635840 INFO  (zkCallback-974-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1636739 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42574_
   [junit4]   2> 1636739 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:42574_
   [junit4]   2> 1636740 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1636740 INFO  (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1636740 INFO  (zkCallback-974-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1636740 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1636740 INFO  (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1636740 INFO  (zkCallback-974-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1636764 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1636764 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1636764 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores
   [junit4]   2> 1636765 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1636765 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.hdfs.HdfsChaosMonkeySafeLeaderTest_29242E7382F60DE5-001/shard-1-001/cores/collection1
   [junit4]   2> 1636765 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1636774 INFO  (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1636774 INFO  (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1636775 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:42574",
   [junit4]   2>   "node_name":"127.0.0.1:42574_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"} current state version: 5
   [junit4]   2> 1636775 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ReplicaMutator Update state numShards=2 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:42574",
   [junit4]   2>   "node_name":"127.0.0.1:42574_",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"collection1",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1636775 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 1636775 INFO  (OverseerStateUpdate-95793786670874628-127.0.0.1:54816_-n_0000000000) [n:127.0.0.1:54816_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1636776 INFO  (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1   x:collection1] o.a.s.c.ZkController Collection collection1 not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 1636776 INFO  (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 1636776 INFO  (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [collection1]
   [junit4]   2> 1636776 INFO  (coreLoadExecutor-2249-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_ c:collection1   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1636779 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 1636779 INFO  (TEST-HdfsChaosMonkeySafeLeaderTest.test-seed#[29242E7382F60DE5]) [n:127.0.0.1:42574_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1636876 INFO  (zkCallback-974-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1636877 INFO  (zkCallback-972-thread-1-processing-n:127.0.0.1:54816_) [n:127.0.0.1:54816_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1636876 INFO  (zkCallback-979-thread-1-processing-n:127.0.0.1:42574_) [n:127.0.0.1:42574_    ] o.a.s.c.c.ZkStateR

[...truncated too long message...]

reAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 1784808 INFO  (SUITE-HdfsChaosMonkeySafeLeaderTest-seed#[29242E7382F60DE5]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]   2> org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException
   [junit4]   2> 	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
   [junit4]   2> 	at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:130)
   [junit4]   2> 	at org.apache.solr.update.HdfsTransactionLog.<init>(HdfsTransactionLog.java:74)
   [junit4]   2> 	at org.apache.solr.update.HdfsUpdateLog.ensureLog(HdfsUpdateLog.java:308)
   [junit4]   2> 	at org.apache.solr.update.UpdateLog.deleteByQuery(UpdateLog.java:490)
   [junit4]   2> 	at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:467)
   [junit4]   2> 	at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:79)
   [junit4]   2> 	at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:52)
   [junit4]   2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:943)
   [junit4]   2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:1438)
   [junit4]   2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1331)
   [junit4]   2> 	at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1140)
   [junit4]   2> 	at org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:124)
   [junit4]   2> 	at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:159)
   [junit4]   2> 	at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:112)
   [junit4]   2> 	at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:54)
   [junit4]   2> 	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:97)
   [junit4]   2> 	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:69)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2015)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:399)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:518)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
   [junit4]   2> 	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> Apr 26, 2016 7:02:24 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=SimpleText, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=fr-BE, timezone=America/Indiana/Vevay
   [junit4]   2> NOTE: Linux 3.13.0-85-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=269967712,total=524812288
   [junit4]   2> NOTE: All tests run in this JVM: [TestXIncludeConfig, LukeRequestHandlerTest, TestSchemaSimilarityResource, TestIntervalFaceting, DebugComponentTest, HdfsSyncSliceTest, DistributedVersionInfoTest, XmlUpdateRequestHandlerTest, StatsComponentTest, TestDocBasedVersionConstraints, TestWriterPerf, RequestHandlersTest, SparseHLLTest, TestCollationField, FullSolrCloudDistribCmdsTest, CurrencyFieldXmlFileTest, ConnectionManagerTest, TestLeaderElectionZkExpiry, DocValuesMultiTest, TestSolrCLIRunExample, TestCharFilters, SearchHandlerTest, DeleteInactiveReplicaTest, BaseCdcrDistributedZkTest, SuggesterTest, TestSolrIndexConfig, TestCloudManagedSchema, ClassificationUpdateProcessorFactoryTest, OverseerTest, CdcrRequestHandlerTest, DistributedFacetPivotWhiteBoxTest, SimpleFacetsTest, DistributedSuggestComponentTest, TlogReplayBufferedWhileIndexingTest, TestIndexingPerformance, ShardSplitTest, CollectionsAPISolrJTest, HdfsChaosMonkeySafeLeaderTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsChaosMonkeySafeLeaderTest -Dtests.seed=29242E7382F60DE5 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=fr-BE -Dtests.timezone=America/Indiana/Vevay -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | HdfsChaosMonkeySafeLeaderTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([29242E7382F60DE5]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [150/591 (1!)] on J2 in 172.03s, 1 test, 1 failure <<< FAILURES!

[...truncated 1401 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 591 suites (10 ignored), 2381 tests, 1 suite-level error, 31 ignored (16 assumptions) [seed: 29242E7382F60DE5]

Total time: 215 minutes 46 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 48 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/48/

No tests ran.

Build Log:
[...truncated 10 lines...]
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://git-wip-us.apache.org/repos/asf/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
	at hudson.scm.SCM.checkout(SCM.java:485)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1738)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
stdout: 
stderr: fatal: Could not write new index file.

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1669)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1665)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1307)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:371)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:669)
	at hudson.plugins.git.GitAPI.clean(GitAPI.java:311)
	at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to lucene(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
	at hudson.remoting.Channel.call(Channel.java:781)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
	at com.sun.proxy.$Proxy115.clean(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:453)
	at hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:32)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:762)
	... 11 more
ERROR: null
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://git-wip-us.apache.org/repos/asf/lucene-solr.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://git-wip-us.apache.org/repos/asf/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
	at hudson.scm.SCM.checkout(SCM.java:485)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1738)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
stdout: 
stderr: fatal: Could not write new index file.

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1669)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1665)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1307)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:371)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:669)
	at hudson.plugins.git.GitAPI.clean(GitAPI.java:311)
	at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to lucene(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
	at hudson.remoting.Channel.call(Channel.java:781)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
	at com.sun.proxy.$Proxy115.clean(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:453)
	at hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:32)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:762)
	... 11 more
ERROR: null
Retrying after 10 seconds
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://git-wip-us.apache.org/repos/asf/lucene-solr.git # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://git-wip-us.apache.org/repos/asf/lucene-solr.git
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:766)
	at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1022)
	at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1053)
	at hudson.scm.SCM.checkout(SCM.java:485)
	at hudson.model.AbstractProject.checkout(AbstractProject.java:1276)
	at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:607)
	at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:529)
	at hudson.model.Run.execute(Run.java:1738)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)
Caused by: hudson.plugins.git.GitException: Command "git reset --hard" returned status code 128:
stdout: 
stderr: fatal: Could not write new index file.

	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1693)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1669)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:1665)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1307)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.reset(CliGitAPIImpl.java:371)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.clean(CliGitAPIImpl.java:669)
	at hudson.plugins.git.GitAPI.clean(GitAPI.java:311)
	at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.perform(RemoteInvocationHandler.java:608)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:583)
	at hudson.remoting.RemoteInvocationHandler$RPCRequest.call(RemoteInvocationHandler.java:542)
	at hudson.remoting.UserRequest.perform(UserRequest.java:120)
	at hudson.remoting.UserRequest.perform(UserRequest.java:48)
	at hudson.remoting.Request$2.run(Request.java:326)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	at ......remote call to lucene(Native Method)
	at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1416)
	at hudson.remoting.UserResponse.retrieve(UserRequest.java:220)
	at hudson.remoting.Channel.call(Channel.java:781)
	at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:250)
	at com.sun.proxy.$Proxy115.clean(Unknown Source)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl.clean(RemoteGitImpl.java:453)
	at hudson.plugins.git.extensions.impl.CleanBeforeCheckout.decorateFetchCommand(CleanBeforeCheckout.java:32)
	at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:762)
	... 11 more
ERROR: null
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: Test reports were found but none of them are new. Did tests run? 
For example, /home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/common/test/TEST-org.apache.lucene.analysis.ar.TestArabicAnalyzer.xml is 9 hr 54 min old

Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 47 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/47/

5 tests failed.
FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556:83BF5F1FE3012393]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh(ZkStateReaderTest.java:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ObjectTracker found 4 object(s) that were not released!!! [MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, RawDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, RawDirectoryWrapper]
	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
	at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556]:0)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556:F74A8E954A6E386C]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11721 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateReaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_9C052EE83361E556-001/init-core-data-001
   [junit4]   2> 1733002 INFO  (SUITE-ZkStateReaderTest-seed#[9C052EE83361E556]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1733012 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> 1733012 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1733012 INFO  (Thread-39852) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1733012 INFO  (Thread-39852) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1733113 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer start zk server on port:44707
   [junit4]   2> 1733113 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1733127 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1733144 INFO  (zkCallback-755-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@85779a8 name:ZooKeeperConnection Watcher:127.0.0.1:44707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1733144 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1733145 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1733155 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1733175 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1733183 INFO  (zkCallback-756-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@72f1d101 name:ZooKeeperConnection Watcher:127.0.0.1:44707 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1733184 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1733184 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1733184 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1733197 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1733208 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1733209 INFO  (zkCallback-757-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2882a6df name:ZooKeeperConnection Watcher:127.0.0.1:44707/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1733209 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1733209 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1733211 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1733211 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1733212 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1733212 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1733213 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1733213 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1733214 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1733214 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 1733215 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 1733215 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 1733217 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 1733220 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1116527757@5fe63837 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@5bd9a713[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 1733220 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1116527757@41c6b760 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@5bd9a713[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 1733220 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1116527757@39f8d322 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@5bd9a713[Shutting down, pool size = 1, active threads = 0, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 1733224 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44707 44707
   [junit4]   2> 1733494 INFO  (Thread-39852) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44707 44707
   [junit4]   2> 1733500 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateReaderTest -Dtests.method=testStateFormatUpdateWithExplicitRefresh -Dtests.seed=9C052EE83361E556 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=hu-HU -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.50s J0 | ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not find collection : c1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556:83BF5F1FE3012393]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh(ZkStateReaderTest.java:42)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1733514 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWatchedCollectionCreation
   [junit4]   2> 1733514 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1733514 INFO  (Thread-39853) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1733514 INFO  (Thread-39853) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1733667 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer start zk server on port:51132
   [junit4]   2> 1733668 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1733687 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1733696 INFO  (zkCallback-758-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d88a032 name:ZooKeeperConnection Watcher:127.0.0.1:51132 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1733696 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1733696 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1733705 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1733708 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1733710 INFO  (zkCallback-759-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@55ff81ca name:ZooKeeperConnection Watcher:127.0.0.1:51132 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1733710 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1733710 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1733710 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1733713 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1733719 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1733727 INFO  (zkCallback-760-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@762f49a1 name:ZooKeeperConnection Watcher:127.0.0.1:51132/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1733728 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1733728 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1733728 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1733729 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1733729 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1733730 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1733732 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1733732 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1733733 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1733733 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 1733733 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 1733733 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 1733735 INFO  (zkCallback-760-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1733741 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 1733741 INFO  (zkCallback-760-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1733741 INFO  (zkCallback-760-thread-1) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 1733800 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51132 51132
   [junit4]   2> 1734296 INFO  (Thread-39853) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51132 51132
   [junit4]   2> 1734297 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWatchedCollectionCreation
   [junit4]   2> 1734299 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelay
   [junit4]   2> 1734300 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1734307 INFO  (Thread-39854) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1734307 INFO  (Thread-39854) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1734543 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer start zk server on port:40859
   [junit4]   2> 1734544 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1734563 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1734579 INFO  (zkCallback-761-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@31f31447 name:ZooKeeperConnection Watcher:127.0.0.1:40859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1734580 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1734580 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1734589 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1734596 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1734596 INFO  (zkCallback-762-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6393a418 name:ZooKeeperConnection Watcher:127.0.0.1:40859 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1734596 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1734597 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1734597 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1734619 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1734621 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1734621 INFO  (zkCallback-763-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@52768869 name:ZooKeeperConnection Watcher:127.0.0.1:40859/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1734621 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1734621 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1734621 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1734622 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1734622 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1734623 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1734623 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1734624 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1734624 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1734624 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 1734625 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 1734625 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 1734626 INFO  (zkCallback-763-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1734628 INFO  (zkCallback-763-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1734676 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 1734677 INFO  (zkCallback-763-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1734677 INFO  (zkCallback-763-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1734677 INFO  (zkCallback-763-thread-3) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 1734684 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40859 40859
   [junit4]   2> 1734799 INFO  (Thread-39854) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40859 40859
   [junit4]   2> 1734800 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelay
   [junit4]   2> 1734802 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> 1734803 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1734812 INFO  (Thread-39855) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1734812 INFO  (Thread-39855) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1734912 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer start zk server on port:54791
   [junit4]   2> 1734912 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1734932 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1734934 INFO  (zkCallback-764-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@491aeab name:ZooKeeperConnection Watcher:127.0.0.1:54791 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1734934 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1734934 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1734935 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1734936 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1734936 INFO  (zkCallback-765-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@55cf2980 name:ZooKeeperConnection Watcher:127.0.0.1:54791 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1734936 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1734937 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1734937 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1734939 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1734939 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1734953 INFO  (zkCallback-766-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@645b867e name:ZooKeeperConnection Watcher:127.0.0.1:54791/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1734953 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1734954 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1734954 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1734954 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1734955 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1734955 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1734956 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1734956 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1734956 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1734964 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 1734965 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 1734976 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/1116527757@572bb8ff rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@12293c7e[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 1734976 INFO  (zkCallback-766-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1734980 WARN  (zkCallback-766-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 1734989 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54791 54791
   [junit4]   2> 1735176 INFO  (Thread-39855) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:54791 54791
   [junit4]   2> 1735183 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateReaderTest -Dtests.method=testStateFormatUpdateWithExplicitRefreshLazy -Dtests.seed=9C052EE83361E556 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=hu-HU -Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.38s J0 | ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not find collection : c1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556:F74A8E954A6E386C]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:47)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1735196 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 1735196 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1735208 INFO  (Thread-39856) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1735208 INFO  (Thread-39856) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1735304 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer start zk server on port:35405
   [junit4]   2> 1735304 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1735305 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1735313 INFO  (zkCallback-767-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@67aa110f name:ZooKeeperConnection Watcher:127.0.0.1:35405 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1735313 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1735313 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1735317 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1735317 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1735318 INFO  (zkCallback-768-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3e5f555c name:ZooKeeperConnection Watcher:127.0.0.1:35405 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1735318 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1735318 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1735318 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1735320 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1735320 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1735321 INFO  (zkCallback-769-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5dd110b1 name:ZooKeeperConnection Watcher:127.0.0.1:35405/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1735321 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1735322 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1735322 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1735322 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1735323 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1735323 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1735324 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1735324 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1735325 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1735325 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 1735326 INFO  (zkCallback-769-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1735326 INFO  (zkCallback-769-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1735327 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 1735327 INFO  (zkCallback-769-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1735329 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35405 35405
   [junit4]   2> 1735675 INFO  (Thread-39856) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35405 35405
   [junit4]   2> 1735684 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 1735686 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalCollectionWatchedNotWatched
   [junit4]   2> 1735686 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1735688 INFO  (Thread-39857) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1735688 INFO  (Thread-39857) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1735789 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer start zk server on port:35435
   [junit4]   2> 1735789 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1735793 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1735815 INFO  (zkCallback-770-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@74872986 name:ZooKeeperConnection Watcher:127.0.0.1:35435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1735815 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1735815 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1735824 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1735827 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1544c20fc850000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1735828 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1735859 INFO  (zkCallback-771-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4ceab78c name:ZooKeeperConnection Watcher:127.0.0.1:35435 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1735860 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1735860 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1735860 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1735907 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1735909 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1735910 INFO  (zkCallback-772-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3698c7b8 name:ZooKeeperConnection Watcher:127.0.0.1:35435/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1735910 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1735910 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1735910 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1735910 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 1735911 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1735911 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1735912 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1735912 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1735912 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1735913 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 1735913 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 1735914 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 1735914 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 1735914 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.c.ZkStateReader Removing watch for uninteresting collection [c1]
   [junit4]   2> 1735914 INFO  (zkCallback-772-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 1735915 WARN  (zkCallback-772-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 1735918 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35435 35435
   [junit4]   2> 1736192 INFO  (Thread-39857) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35435 35435
   [junit4]   2> 1736193 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalCollectionWatchedNotWatched
   [junit4]   2> 1736194 INFO  (SUITE-ZkStateReaderTest-seed#[9C052EE83361E556]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_9C052EE83361E556-001
   [junit4]   2> Apr 25, 2016 6:35:11 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=131, maxMBSortInHeap=5.360787588278357, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=hu-HU, timezone=SystemV/EST5
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=196202728,total=534773760
   [junit4]   2> NOTE: All tests run in this JVM: [QueryEqualityTest, BlockDirectoryTest, TestLMDirichletSimilarityFactory, TestIBSimilarityFactory, PeerSyncTest, StandardRequestHandlerTest, CollectionTooManyReplicasTest, TestSizeLimitedDistributedMap, TestAnalyzedSuggestions, LeaderInitiatedRecoveryOnCommitTest, TriLevelCompositeIdRoutingTest, TestRandomRequestDistribution, TestMacros, TestLRUStatsCache, HdfsBasicDistributedZkTest, TestManagedSynonymFilterFactory, SynonymTokenizerTest, BitVectorTest, AddSchemaFieldsUpdateProcessorFactoryTest, DateFieldTest, TestDistribIDF, TestTolerantUpdateProcessorCloud, BlockJoinFacetRandomTest, EchoParamsTest, TestConfigSets, TestRemoteStreaming, CloneFieldUpdateProcessorFactoryTest, TestFastOutputStream, URLClassifyProcessorTest, TestSchemaNameResource, PreAnalyzedFieldTest, TestRequestForwarding, DateRangeFieldTest, HdfsRecoveryZkTest, SpatialRPTFieldTypeTest, BasicDistributedZkTest, TestAuthorizationFramework, TestSolr4Spatial, TestDocSet, FacetPivotSmallTest, TestStressLucene, PingRequestHandlerTest, DeleteLastCustomShardedReplicaTest, ActionThrottleTest, UUIDFieldTest, DistributedExpandComponentTest, RankQueryTest, SolrRequestParserTest, TestRealTimeGet, TestFoldingMultitermQuery, ZkStateReaderTest]
   [junit4] Completed [141/591 (1!)] on J0 in 4.03s, 6 tests, 2 errors <<< FAILURES!

[...truncated 712 lines...]
   [junit4] Suite: org.apache.solr.core.TestLazyCores
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/init-core-data-001
   [junit4]   2> 4373684 INFO  (SUITE-TestLazyCores-seed#[9C052EE83361E556]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 4373692 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testMidUseUnload
   [junit4]   2> 4373705 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001'
   [junit4]   2> 4373705 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4373705 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 4373705 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreContainer New CoreContainer 712820093
   [junit4]   2> 4373705 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001]
   [junit4]   2> 4373706 WARN  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/lib
   [junit4]   2> 4373706 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 4373707 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=600000&connTimeout=60000&retry=true
   [junit4]   2> 4373707 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.SolrCores Allocating transient cache for 4 transient cores
   [junit4]   2> 4373710 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4373710 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4373710 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 4373710 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 4373711 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 4373711 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection2, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 4373711 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection3, config=solrconfig.xml, loadOnStartup=false, schema=schema.xml, configSetProperties=configsetprops.json, transient=on, dataDir=data/}
   [junit4]   2> 4373713 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection4, config=solrconfig.xml, loadOnStartup=false, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 4373713 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection5, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 4373713 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection6, config=solrconfig.xml, loadOnStartup=false, schema=schema.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 4373713 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection7, config=solrconfig.xml, loadOnStartup=false, schema=schema.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 4373714 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection8, config=solrconfig.xml, loadOnStartup=false, schema=schema.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 4373714 INFO  (TEST-TestLazyCores.testMidUseUnload-seed#[9C052EE83361E556]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection9, config=solrconfig.xml, loadOnStartup=false, schema=schema.xml, configSetProperties=configsetprops.json, transient=true, dataDir=data/}
   [junit4]   2> 4373714 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1'
   [junit4]   2> 4373716 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4373716 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 4373718 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection5'
   [junit4]   2> 4373718 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4373718 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 4373719 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2'
   [junit4]   2> 4373720 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4373720 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 4373724 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 4373724 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 4373730 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 4373730 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 4373735 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 4373735 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 4373739 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4373741 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.s.IndexSchema [collection2] Schema name=tiny
   [junit4]   2> 4373747 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 4373748 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4373749 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2'
   [junit4]   2> 4373749 INFO  (coreLoadExecutor-11259-thread-2) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection2' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2
   [junit4]   2> 4373749 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4373749 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore [[collection2] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data/]
   [junit4]   2> 4373749 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore JMX monitoring not detected for core: collection2
   [junit4]   2> 4373750 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data
   [junit4]   2> 4373750 INFO  (coreLoadExecutor-11259-thread-3) [    ] o.a.s.s.IndexSchema [collection5] Schema name=tiny
   [junit4]   2> 4373750 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data/index/
   [junit4]   2> 4373750 WARN  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore [collection2] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4373751 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data/index
   [junit4]   2> 4373751 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@73246925
   [junit4]   2> 4373817 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4373819 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=tiny
   [junit4]   2> 4373823 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6e006005 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@763d086b),segFN=segments_1,generation=1}
   [junit4]   2> 4373824 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 4373826 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 4373827 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1'
   [junit4]   2> 4373827 INFO  (coreLoadExecutor-11259-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1
   [junit4]   2> 4373827 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 4373827 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1/data/]
   [junit4]   2> 4373827 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 4373828 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1/data
   [junit4]   2> 4373828 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1/data/index/
   [junit4]   2> 4373828 WARN  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4373831 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4373841 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4373842 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4373842 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4373843 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4373844 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/select,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 4373844 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 4373844 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection1/data/index
   [junit4]   2> 4373845 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3d3b900f
   [junit4]   2> 4373845 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@3e79605f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5c293f82),segFN=segments_1,generation=1}
   [junit4]   2> 4373845 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 4373847 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4373848 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4373848 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@4cbd3ad1
   [junit4]   2> 4373848 INFO  (OldIndexDirectoryCleanupThreadForCore-collection2) [    x:collection2] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection2 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data/
   [junit4]   2> 4373848 WARN  (OldIndexDirectoryCleanupThreadForCore-collection2) [    x:collection2] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 4373848 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@6e006005 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@763d086b),segFN=segments_1,generation=1}
   [junit4]   2> 4373848 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 4373848 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.s.SolrIndexSearcher Opening [Searcher@2cb503f1[collection2] main]
   [junit4]   2> 4373848 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/conf
   [junit4]   2> 4373849 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/conf}
   [junit4]   2> 4373849 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.core.TestLazyCores_9C052EE83361E556-001/tempDir-001/collection2/conf
   [junit4]   2> 4373849 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 4373849 INFO  (coreLoadExecutor-11259-thread-2) [    x:collection2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4373855 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4373855 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4373856 INFO  (coreLoadExecutor-11259-thread-1) [    x:collection1] o.a.

[...truncated too long message...]

039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Apr 25, 2016 7:21:36 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]   2> Apr 25, 2016 7:21:39 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=2, maxDocsPerChunk=9, blockSize=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=2, blockSize=7)), sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=en, timezone=America/Thunder_Bay
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=166956272,total=533200896
   [junit4]   2> NOTE: All tests run in this JVM: [TestMaxScoreQueryParser, TestIndexingPerformance, SuggesterTSTTest, TestSuggestSpellingConverter, ClassificationUpdateProcessorFactoryTest, CoreAdminHandlerTest, FullSolrCloudDistribCmdsTest, UnloadDistributedZkTest, DirectUpdateHandlerTest, RemoteQueryErrorTest, HdfsChaosMonkeySafeLeaderTest, TestZkChroot, TestPHPSerializedResponseWriter, TestFileDictionaryLookup, HdfsSyncSliceTest, MergeStrategyTest, FullHLLTest, RegexBoostProcessorTest, TestRecovery, CurrencyFieldXmlFileTest, SuggesterWFSTTest, TestStressLiveNodes, HdfsBasicDistributedZk2Test, TestRuleBasedAuthorizationPlugin, TestGroupingSearch, TestSolrIndexConfig, TestIntervalFaceting, TestCustomDocTransformer, ScriptEngineTest, TestSolrCLIRunExample, DistanceUnitsTest, TestSchemaSimilarityResource, TestCollationFieldDocValues, TlogReplayBufferedWhileIndexingTest, DocValuesMultiTest, TestClusterStateMutator, AnalyticsMergeStrategyTest, ConnectionManagerTest, TestFuzzyAnalyzedSuggestions, TestJsonRequest, TestSearchPerf, JavabinLoaderTest, TestPerFieldSimilarity, CoreAdminCreateDiscoverTest, MigrateRouteKeyTest, CollectionStateFormat2Test, TestFilteredDocIdSet, SearchHandlerTest, TestOverriddenPrefixQueryForCustomFieldType, AnalysisErrorHandlingTest, BlockJoinFacetDistribTest, TestSmileRequest, DirectoryFactoryTest, ClusterStateTest, SimpleCollectionCreateDeleteTest, TestOnReconnectListenerSupport, LeaderFailoverAfterPartitionTest, BaseCdcrDistributedZkTest, TestElisionMultitermQuery, DistributedDebugComponentTest, TestNRTOpen, TestManagedResource, FieldAnalysisRequestHandlerTest, TestDefaultSearchFieldResource, TestSolrDeletionPolicy1, TestWriterPerf, HdfsRecoverLeaseTest, TestCharFilters, TestBulkSchemaAPI, SolrCoreCheckLockOnStartupTest, DebugComponentTest, XmlUpdateRequestHandlerTest, TestRawTransformer, RollingRestartTest, DateMathParserTest, TestFieldTypeResource, InfoHandlerTest, BigEndianAscendingWordSerializerTest, TestJettySolrRunner, HdfsUnloadDistributedZkTest, TestJmxMonitoredMap, BadComponentTest, TestCloudManagedSchema, OverseerStatusTest, TestStressRecovery, TestDefaultStatsCache, TestFieldCollectionResource, TestIndexSearcher, PKIAuthenticationIntegrationTest, DistributedFacetPivotWhiteBoxTest, TestAuthenticationFramework, OverriddenZkACLAndCredentialsProvidersTest, TestExpandComponent, TestPerFieldSimilarityClassic, DistribDocExpirationUpdateProcessorTest, SparseHLLTest, CheckHdfsIndexTest, AutoCommitTest, TestDistributedGrouping, TestFaceting, TermVectorComponentDistributedTest, TestStressReorder, TestJoin, TestReload, TestStressVersions, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, SimpleFacetsTest, SolrCoreTest, StatsComponentTest, SpellCheckComponentTest, QueryElevationComponentTest, BadIndexSchemaTest, ConvertedLegacyTest, TestSort, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLazyCores -Dtests.seed=9C052EE83361E556 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en -Dtests.timezone=America/Thunder_Bay -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestLazyCores (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MDCAwareThreadPoolExecutor, MockDirectoryWrapper, SolrCore, RawDirectoryWrapper]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   [junit4]    >    1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=113140, name=searcherExecutor-11457-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([9C052EE83361E556]:0)
   [junit4] Completed [344/591 (2!)] on J2 in 150.08s, 8 tests, 1 failure, 2 errors <<< FAILURES!

[...truncated 745 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 591 suites (10 ignored), 2381 tests, 3 suite-level errors, 2 errors, 804 ignored (16 assumptions) [seed: 9C052EE83361E556]

Total time: 347 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 46 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/46/

4 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:34956

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:34956
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:457)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([E4A8AA33FD235588:8FE70A4E842C88B2]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:47)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.core.TestDynamicLoading.testDynamicLoading

Error Message:
Could not get expected value  'X val' for path 'x' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "params":{"wt":"json"},   "context":{     "webapp":"/izcsw/fi",     "path":"/test1",     "httpMethod":"GET"},   "class":"org.apache.solr.core.BlobStoreTestRequestHandler",   "x":null},  from server:  null

Stack Trace:
java.lang.AssertionError: Could not get expected value  'X val' for path 'x' full output: {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "params":{"wt":"json"},
  "context":{
    "webapp":"/izcsw/fi",
    "path":"/test1",
    "httpMethod":"GET"},
  "class":"org.apache.solr.core.BlobStoreTestRequestHandler",
  "x":null},  from server:  null
	at __randomizedtesting.SeedInfo.seed([E4A8AA33FD235588:3CE587640AFEF028]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:457)
	at org.apache.solr.core.TestDynamicLoading.testDynamicLoading(TestDynamicLoading.java:233)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
ObjectTracker found 2 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper]
	at __randomizedtesting.SeedInfo.seed([E4A8AA33FD235588]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
	at sun.reflect.GeneratedMethodAccessor26.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12585 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/init-core-data-001
   [junit4]   2> 4984762 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 4984763 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 4984879 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 4984886 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4984888 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4984902 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_53376_hdfs____.w22kpu/webapp
   [junit4]   2> 4985313 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 4985752 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:53376
   [junit4]   2> 4986124 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4986124 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4986139 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34884_datanode____.vwciu/webapp
   [junit4]   2> 4986220 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 4986716 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34884
   [junit4]   2> 4986786 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 4986787 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 4986801 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_52885_datanode____.f4brfl/webapp
   [junit4]   2> 4986860 INFO  (IPC Server handler 3 on 46173) [    ] BlockStateChange BLOCK* processReport: from storage DS-05c397af-2aed-495d-8940-f7529ea12c16 node DatanodeRegistration(127.0.0.1, datanodeUuid=491bf2dc-de19-4a9a-9e08-59b33e2919cb, infoPort=34884, ipcPort=58446, storageInfo=lv=-56;cid=testClusterID;nsid=1601857524;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 4986860 INFO  (IPC Server handler 3 on 46173) [    ] BlockStateChange BLOCK* processReport: from storage DS-78c9130c-bc66-4a0b-b96e-b175b3ef262c node DatanodeRegistration(127.0.0.1, datanodeUuid=491bf2dc-de19-4a9a-9e08-59b33e2919cb, infoPort=34884, ipcPort=58446, storageInfo=lv=-56;cid=testClusterID;nsid=1601857524;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 4986904 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 4987407 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E4A8AA33FD235588]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:52885
   [junit4]   2> 4987765 INFO  (IPC Server handler 5 on 46173) [    ] BlockStateChange BLOCK* processReport: from storage DS-baddebc8-147d-4674-8f2e-c4c141e5a61c node DatanodeRegistration(127.0.0.1, datanodeUuid=afdea3b6-b01d-437f-9ddf-31f9014a98f0, infoPort=52885, ipcPort=45964, storageInfo=lv=-56;cid=testClusterID;nsid=1601857524;c=0), blocks: 0, hasStaleStorages: true, processing time: 7 msecs
   [junit4]   2> 4987768 INFO  (IPC Server handler 5 on 46173) [    ] BlockStateChange BLOCK* processReport: from storage DS-127a36df-b438-43e6-9824-c6268bcc922b node DatanodeRegistration(127.0.0.1, datanodeUuid=afdea3b6-b01d-437f-9ddf-31f9014a98f0, infoPort=52885, ipcPort=45964, storageInfo=lv=-56;cid=testClusterID;nsid=1601857524;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 4987917 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 4987927 INFO  (Thread-14776) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 4987928 INFO  (Thread-14776) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 4988034 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.ZkTestServer start zk server on port:40789
   [junit4]   2> 4988034 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 4988035 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988073 INFO  (zkCallback-4297-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@132e80f4 name:ZooKeeperConnection Watcher:127.0.0.1:40789 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988073 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988074 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 4988074 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 4988078 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 4988078 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988079 INFO  (zkCallback-4298-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@70c813c name:ZooKeeperConnection Watcher:127.0.0.1:40789/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988080 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988080 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 4988080 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 4988082 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 4988083 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 4988084 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 4988085 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 4988085 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 4988087 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 4988087 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 4988089 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4988089 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4988090 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 4988090 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 4988091 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 4988091 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 4988092 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 4988092 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 4988093 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 4988094 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 4988095 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4988095 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 4988096 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4988096 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4988097 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 4988097 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 4988098 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 4988098 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 4988100 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 4988100 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988101 INFO  (zkCallback-4299-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@30fc3413 name:ZooKeeperConnection Watcher:127.0.0.1:40789 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988101 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988101 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 4988101 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 4988102 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 4988103 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988103 INFO  (zkCallback-4300-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1963957f name:ZooKeeperConnection Watcher:127.0.0.1:40789/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988104 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988104 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 4988104 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 4988104 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 4988106 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 4988106 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 4988107 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 4988107 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 4988108 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4988108 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 4988109 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 4988109 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 4988110 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 4988110 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 4988111 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 4988111 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 4988113 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 4988113 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 4988114 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4988114 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 4988115 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 4988115 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 4988116 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 4988116 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 4988117 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 4988117 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 4988184 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores/collection1
   [junit4]   2> 4988185 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 4988240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@ff1933a{/,null,AVAILABLE}
   [junit4]   2> 4988240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4e200314{HTTP/1.1,[http/1.1]}{127.0.0.1:53428}
   [junit4]   2> 4988240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.e.j.s.Server Started @4993601ms
   [junit4]   2> 4988240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:46173/hdfs__localhost_46173__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001_tempDir-002_control_data, hostContext=/, hostPort=53428, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores}
   [junit4]   2> 4988240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 4988240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001'
   [junit4]   2> 4988241 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4988241 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 4988241 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 4988248 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988267 INFO  (zkCallback-4301-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5aeba3aa name:ZooKeeperConnection Watcher:127.0.0.1:40789/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988268 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988270 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 4988271 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 4988271 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/solr.xml
   [junit4]   2> 4988274 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores
   [junit4]   2> 4988274 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.CoreContainer New CoreContainer 649673635
   [junit4]   2> 4988274 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001]
   [junit4]   2> 4988275 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/lib
   [junit4]   2> 4988275 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 4988276 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 4988276 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 4988276 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 4988276 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:40789/solr
   [junit4]   2> 4988276 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 4988276 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 4988277 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988277 INFO  (zkCallback-4304-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6f7696ef name:ZooKeeperConnection Watcher:127.0.0.1:40789 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988277 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988277 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 4988278 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$27/1254703706@5a9bee97
   [junit4]   2> 4988279 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 4988280 INFO  (zkCallback-4305-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@649e18a5 name:ZooKeeperConnection Watcher:127.0.0.1:40789/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 4988280 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 4988280 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 4988281 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 4988282 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 4988283 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 4988284 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 4988284 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 4988285 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 4988285 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 4988286 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 4988286 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 4988287 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 4988287 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 4988288 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 4988288 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 4988288 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95777702432014342-127.0.0.1:53428_-n_0000000000
   [junit4]   2> 4988289 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53428_
   [junit4]   2> 4988289 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 4988289 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.Overseer Overseer (id=95777702432014342-127.0.0.1:53428_-n_0000000000) starting
   [junit4]   2> 4988290 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 4988292 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 4988293 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 4988294 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 4988294 INFO  (OverseerCollectionConfigSetProcessor-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 4988295 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53428_
   [junit4]   2> 4988295 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:53428_
   [junit4]   2> 4988296 INFO  (zkCallback-4305-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 4988296 INFO  (zkCallback-4305-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 4988309 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 4988309 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 4988310 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores
   [junit4]   2> 4988310 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 4988310 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores/collection1
   [junit4]   2> 4988310 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 4988311 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 4988311 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 4988312 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection control_collection not visible yet, but flagging it so a watch is registered when it becomes visible
   [junit4]   2> 4988312 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [control_collection]
   [junit4]   2> 4988312 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Deleting data for [control_collection]
   [junit4]   2> 4988312 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 4988313 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 4988313 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E4A8AA33FD235588]) [n:127.0.0.1:53428_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 4988313 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:53428",
   [junit4]   2>   "node_name":"127.0.0.1:53428_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 4988313 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:53428",
   [junit4]   2>   "node_name":"127.0.0.1:53428_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 4988313 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 4988314 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 4988314 INFO  (zkCallback-4305-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores/collection1'
   [junit4]   2> 4989313 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 4989314 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 4989314 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 4989317 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 4989322 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 4989329 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 4989333 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 4989612 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 4989618 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 4989619 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 4989625 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 4989627 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 4989628 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 4989629 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 4989629 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 4989629 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 4989630 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 4989630 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:46173/solr_hdfs_home
   [junit4]   2> 4989630 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 4989630 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E4A8AA33FD235588-001/control-001/cores/collection1], dataDir=[hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 4989630 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6d35d731
   [junit4]   2> 4989630 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 4989656 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 4989661 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 4989679 WARN  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:46173/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 4989680 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 4989714 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 4989714 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=5, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 4989896 INFO  (IPC Server handler 1 on 46173) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:37164 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-baddebc8-147d-4674-8f2e-c4c141e5a61c:NORMAL:127.0.0.1:37164|RBW], ReplicaUnderConstruction[[DISK]DS-78c9130c-bc66-4a0b-b96e-b175b3ef262c:NORMAL:127.0.0.1:43736|RBW]]} size 0
   [junit4]   2> 4989899 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39b6f140; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 4989899 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 4989899 INFO  (IPC Server handler 3 on 46173) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:43736 is added to blk_1073741825_1001 size 71
   [junit4]   2> 4989905 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 4989907 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 4989907 WARN  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 4989909 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4989909 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4989910 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4989910 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 4989912 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 4989912 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 4989913 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 4989913 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 4989913 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 4989920 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 4989921 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 4989924 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 4989928 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:46173/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@39b6f140; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 4989928 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6633192a[collection1] main]
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 4989929 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4989930 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 4989930 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 4989930 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 4989930 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 4989931 INFO  (searcherExecutor-9862-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@6633192a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 4989931 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 4989931 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 4989931 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 4989931 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1532443241008332800
   [junit4]   2> 4989933 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1532443241008332800
   [junit4]   2> 4989933 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 4989933 INFO  (coreLoadExecutor-9861-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 4989933 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:53428 collection:control_collection shard:shard1
   [junit4]   2> 4989934 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 4989935 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95777702432014342-core_node1-n_0000000000
   [junit4]   2> 4989935 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 4989937 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 4989937 INFO  (zkCallback-4305-thread-1-processing-n:127.0.0.1:53428_) [n:127.0.0.1:53428_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 4989939 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 4989939 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 4989939 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:53428/collection1/
   [junit4]   2> 4989939 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 4989939 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:53428/collection1/ has no replicas
   [junit4]   2> 4989941 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 4989942 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95777702432014342-core_node1-n_0000000000
   [junit4]   2> 4989943 INFO  (OverseerStateUpdate-95777702432014342-127.0.0.1:53428_-n_0000000000) [n:127.0.0.1:53428_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:53428",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 4989944 INFO  (coreZkRegister-9855-thread-1-processing-n:127.0.0.1:53428_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:53428_ c:control_collection s:shard1 r:core_node1 x:collect

[...truncated too long message...]

hread.java:745)
   [junit4]   2> 
   [junit4]   2> 6458322 INFO  (Thread-69080) [n:127.0.0.1:47740_solr c:testschemaapi s:shard1 r:core_node2 x:testschemaapi_shard1_replica2] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.schema.TestManagedSchemaAPI_E4A8AA33FD235588-001/tempDir-001/node1/./testschemaapi_shard1_replica2/data/index
   [junit4]   2> 6458344 INFO  (Thread-69080) [n:127.0.0.1:47740_solr c:testschemaapi s:shard1 r:core_node2 x:testschemaapi_shard1_replica2] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.schema.TestManagedSchemaAPI_E4A8AA33FD235588-001/tempDir-001/node1/./testschemaapi_shard1_replica2/data [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.schema.TestManagedSchemaAPI_E4A8AA33FD235588-001/tempDir-001/node1/./testschemaapi_shard1_replica2/data;done=false>>]
   [junit4]   2> 6458344 INFO  (Thread-69080) [n:127.0.0.1:47740_solr c:testschemaapi s:shard1 r:core_node2 x:testschemaapi_shard1_replica2] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=true
   [junit4]   2> 6458344 INFO  (Thread-69080) [n:127.0.0.1:47740_solr c:testschemaapi s:shard1 r:core_node2 x:testschemaapi_shard1_replica2] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.schema.TestManagedSchemaAPI_E4A8AA33FD235588-001/tempDir-001/node1/./testschemaapi_shard1_replica2/data
   [junit4]   2> 6458344 WARN  (Thread-69080) [n:127.0.0.1:47740_solr c:testschemaapi s:shard1 r:core_node2 x:testschemaapi_shard1_replica2] o.a.s.c.ZkController listener throws error
   [junit4]   2> org.apache.solr.common.SolrException: Unable to reload core [testschemaapi_shard1_replica2]
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:921)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$6(SolrCore.java:2488)
   [junit4]   2> 	at org.apache.solr.cloud.ZkController$4.run(ZkController.java:2425)
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Could not find config name for collection:testschemaapi
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:772)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.reload(SolrCore.java:461)
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:915)
   [junit4]   2> 	... 2 more
   [junit4]   2> Caused by: org.apache.solr.common.SolrException: Could not find config name for collection:testschemaapi
   [junit4]   2> 	at org.apache.solr.rest.ManagedResourceStorage.newStorageIO(ManagedResourceStorage.java:101)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.initRestManager(SolrCore.java:2283)
   [junit4]   2> 	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:748)
   [junit4]   2> 	... 4 more
   [junit4]   2> Apr 23, 2016 11:13:48 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=RandomSimilarity(queryNorm=true,coord=yes): {}, locale=ja-JP-u-ca-japanese-x-lvariant-JP, timezone=Asia/Dacca
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=275464768,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [ChangedSchemaMergeTest, TestSolrQueryParser, TestDistribDocBasedVersion, PrimitiveFieldTypeTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestTolerantSearch, TestCustomDocTransformer, MigrateRouteKeyTest, TestSchemaSimilarityResource, TestCharFilters, EnumFieldTest, TestUtils, TestJmxIntegration, TestSolrConfigHandler, PKIAuthenticationIntegrationTest, RollingRestartTest, HardAutoCommitTest, FullHLLTest, UnloadDistributedZkTest, TestLeaderElectionZkExpiry, CoreAdminHandlerTest, DeleteShardTest, OverseerStatusTest, TestDocBasedVersionConstraints, TlogReplayBufferedWhileIndexingTest, FieldMutatingUpdateProcessorTest, TestCloudPivotFacet, BinaryUpdateRequestHandlerTest, TestStressReorder, SolrCoreCheckLockOnStartupTest, SparseHLLTest, TestSearchPerf, RecoveryZkTest, TestDefaultStatsCache, TestSolrJ, HdfsRecoverLeaseTest, TestPostingsSolrHighlighter, TestElisionMultitermQuery, DeleteInactiveReplicaTest, TestJettySolrRunner, LeaderElectionTest, TestMaxScoreQueryParser, TestReloadDeadlock, DocValuesMultiTest, TestMiniSolrCloudClusterBase, ClusterStateTest, TestClusterStateMutator, InfoHandlerTest, SimpleCollectionCreateDeleteTest, SliceStateTest, TestBulkSchemaAPI, TestRuleBasedAuthorizationPlugin, DistributedFacetPivotLongTailTest, DistribDocExpirationUpdateProcessorTest, RegexBoostProcessorTest, SearchHandlerTest, TestReload, VersionInfoTest, TestFastWriter, TermVectorComponentDistributedTest, TestCoreDiscovery, BaseCdcrDistributedZkTest, TestFileDictionaryLookup, TestShortCircuitedRequests, TestComplexPhraseQParserPlugin, TestStandardQParsers, CollectionsAPISolrJTest, HdfsBasicDistributedZk2Test, TestPerFieldSimilarity, CleanupOldIndexTest, DirectUpdateHandlerOptimizeTest, AssignTest, CoreAdminCreateDiscoverTest, TestExpandComponent, TestManagedStopFilterFactory, SuggestComponentContextFilterQueryTest, TestZkChroot, LukeRequestHandlerTest, TestDistributedMissingSort, TestClassicSimilarityFactory, TestSweetSpotSimilarityFactory, NoCacheHeaderTest, DirectSolrConnectionTest, TestNamedUpdateProcessors, TestWriterPerf, StatelessScriptUpdateProcessorFactoryTest, TestManagedResource, CheckHdfsIndexTest, DistributedVersionInfoTest, TestStressLiveNodes, TestOnReconnectListenerSupport, TestRecovery, OverriddenZkACLAndCredentialsProvidersTest, HdfsChaosMonkeySafeLeaderTest, DistributedDebugComponentTest, TestRandomFaceting, ZkSolrClientTest, ZkCLITest, TestDistributedSearch, CollectionStateFormat2Test, CollectionsAPIAsyncDistributedZkTest, ConfigSetsAPITest, DistributedQueueTest, LeaderFailoverAfterPartitionTest, LeaderInitiatedRecoveryOnShardRestartTest, SSLMigrationTest, SaslZkACLProviderTest, ShardSplitTest, HdfsNNFailoverTest, HdfsRestartWhileUpdatingTest, HdfsSyncSliceTest, HdfsUnloadDistributedZkTest, HdfsWriteToMultipleCollectionsTest, StressHdfsTest, ZkStateReaderTest, RuleEngineTest, RulesTest, TestReplicationHandlerBackup, TestSQLHandler, SecurityConfHandlerTest, SegmentsInfoRequestHandlerTest, DistributedFacetPivotSmallAdvancedTest, DistributedFacetPivotWhiteBoxTest, DistributedQueryComponentOptimizationTest, DistributedSuggestComponentTest, FacetPivotSmallTest, SpatialHeatmapFacetsTest, TestPivotHelperCode, TestTrackingShardHandlerFactory, JavabinLoaderTest, WrapperMergePolicyFactoryTest, SmileWriterTest, TestIntervalFaceting, TestMacros, TestChildDocTransformer, TestGeoJSONResponseWriter, TestRawResponseWriter, TestRawTransformer, TestCopyFieldCollectionResource, TestDefaultSearchFieldResource, TestDynamicFieldCollectionResource, TestDynamicFieldResource, TestFieldCollectionResource, TestFieldResource, TestFieldTypeCollectionResource, TestFieldTypeResource, TestSchemaVersionResource, TestSerializedLuceneMatchVersion, TestSolrQueryParserDefaultOperatorResource, DocValuesMissingTest, SpatialRPTFieldTypeTest, TestBulkSchemaConcurrent, TestCloudManagedSchema, TestCloudSchemaless, TestCollationFieldDocValues, TestManagedSchemaAPI]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestManagedSchemaAPI -Dtests.seed=E4A8AA33FD235588 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ja-JP-u-ca-japanese-x-lvariant-JP -Dtests.timezone=Asia/Dacca -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestManagedSchemaAPI (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [RawDirectoryWrapper, RawDirectoryWrapper]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E4A8AA33FD235588]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [527/591 (4!)] on J2 in 15.74s, 1 test, 1 failure <<< FAILURES!

[...truncated 225 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 591 suites (10 ignored), 2381 tests, 1 suite-level error, 2 errors, 1 failure, 31 ignored (16 assumptions) [seed: E4A8AA33FD235588]

Total time: 332 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 45 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/45/

1 tests failed.
FAILED:  org.apache.lucene.search.spans.TestSpanCollection.testOrQuery

Error Message:
Missing term field:w3

Stack Trace:
java.lang.AssertionError: Missing term field:w3
	at __randomizedtesting.SeedInfo.seed([610D9961FC47383B:169D2540D154D73]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.lucene.search.spans.TestSpanCollection.checkCollectedTerms(TestSpanCollection.java:103)
	at org.apache.lucene.search.spans.TestSpanCollection.testOrQuery(TestSpanCollection.java:147)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 1586 lines...]
   [junit4] Suite: org.apache.lucene.search.spans.TestSpanCollection
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestSpanCollection -Dtests.method=testOrQuery -Dtests.seed=610D9961FC47383B -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar-BH -Dtests.timezone=Asia/Aqtau -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] FAILURE 0.10s J1 | TestSpanCollection.testOrQuery <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Missing term field:w3
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([610D9961FC47383B:169D2540D154D73]:0)
   [junit4]    > 	at org.apache.lucene.search.spans.TestSpanCollection.checkCollectedTerms(TestSpanCollection.java:103)
   [junit4]    > 	at org.apache.lucene.search.spans.TestSpanCollection.testOrQuery(TestSpanCollection.java:147)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/test/J1/temp/lucene.search.spans.TestSpanCollection_610D9961FC47383B-001
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=RandomSimilarity(queryNorm=true,coord=yes): {field=DFR GZ(0.3)}, locale=ar-BH, timezone=Asia/Aqtau
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=130024856,total=404750336
   [junit4]   2> NOTE: All tests run in this JVM: [TestSetOnce, TestPerFieldPostingsFormat2, TestRegexpQuery, TestIndexWriterOnVMError, TestSegmentTermEnum, Test2BPoints, Test2BPostings, TestAutomatonQueryUnicode, TestRamUsageEstimator, TestNRTCachingDirectory, TestCompiledAutomaton, TestOmitTf, TestDocumentsWriterStallControl, TestHugeRamFile, TestPositionIncrement, TestRegExp, TestTrackingDirectoryWrapper, TestBooleanScorer, TestDocument, TestParallelTermEnum, TestDocsAndPositions, TestAllFilesCheckIndexHeader, TestParallelLeafReader, TestBooleanQuery, TestSameTokenSamePosition, TestMathUtil, TestIndexWriterNRTIsCurrent, TestTryDelete, TestSimpleFSLockFactory, TestPackedInts, TestSimilarity, TestNumericDocValuesUpdates, TestScoreCachingWrappingScorer, TestFieldValueQuery, Test2BPostingsBytes, TestPerFieldDocValuesFormat, TestDocInverterPerFieldErrorInfo, TestScorerPerf, TestFlex, TestReadOnlyIndex, TestTopFieldCollector, TestCachingTokenFilter, TestFSTs, TestSmallFloat, TestIndexWriterExceptions2, TestVersion, TestDirectoryReaderReopen, TestSortedSetDocValues, TestFieldsReader, TestRollingBuffer, TestStressIndexing, TestGrowableByteArrayDataOutput, TestFileSwitchDirectory, TestIndexWriterOnJRECrash, TestNativeFSLockFactory, TestPayloadsOnVectors, TestBM25Similarity, TestFrequencyTrackingRingBuffer, TestMultiPhraseEnum, TestSearchAfter, TestFieldMaskingSpanQuery, TestBoolean2, TestSpanBoostQuery, TestControlledRealTimeReopenThread, TestSparseFixedBitDocIdSet, TestBagOfPostings, TestAllFilesHaveCodecHeader, TestRecyclingIntBlockAllocator, TestMixedDocValuesUpdates, TestQueryRescorer, TestReqExclBulkScorer, TestSortRescorer, TestSortedNumericSortField, TestSortedSetSortField, TestClassicSimilarity, TestSpanCollection]
   [junit4] Completed [389/418 (1!)] on J1 in 0.19s, 3 tests, 1 failure <<< FAILURES!

[...truncated 113 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 418 suites (1 ignored), 3502 tests, 1 failure, 38 ignored (34 assumptions) [seed: 610D9961FC47383B]

Total time: 85 minutes 11 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 44 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/44/

4 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Captured an uncaught exception in thread: Thread[id=32313, name=Thread-31915, state=RUNNABLE, group=TGRP-TestDistributedSearch]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=32313, name=Thread-31915, state=RUNNABLE, group=TGRP-TestDistributedSearch]
	at __randomizedtesting.SeedInfo.seed([FF7B6F80A8BACE99:772F505A0646A361]:0)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:53032/gr_y/collection1: java.lang.NullPointerException
	at org.apache.solr.search.grouping.distributed.responseprocessor.TopGroupsShardResponseProcessor.process(TopGroupsShardResponseProcessor.java:105)
	at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:753)
	at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:736)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2015)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

	at __randomizedtesting.SeedInfo.seed([FF7B6F80A8BACE99]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.TestDistributedSearch$1.run(TestDistributedSearch.java:1114)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=107907, name=collection2, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=107907, name=collection2, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.common.SolrException: Error reading cluster properties
	at __randomizedtesting.SeedInfo.seed([FF7B6F80A8BACE99]:0)
	at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:769)
	at org.apache.solr.common.cloud.ZkStateReader.getBaseUrlForNodeName(ZkStateReader.java:863)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1090)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:994)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /clusterprops.json
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
	at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1045)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:314)
	at org.apache.solr.common.cloud.SolrZkClient$5.execute(SolrZkClient.java:311)
	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:60)
	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:311)
	at org.apache.solr.common.cloud.ZkStateReader.getClusterProps(ZkStateReader.java:760)
	... 6 more


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([FF7B6F80A8BACE99:E0C11E7778DA085C]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh(ZkStateReaderTest.java:42)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Thu Apr 21 00:20:55 DAVT 2016

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Thu Apr 21 00:20:55 DAVT 2016
	at __randomizedtesting.SeedInfo.seed([FF7B6F80A8BACE99:24D06F46AD92A72A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1423)
	at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:775)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11190 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateReaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_FF7B6F80A8BACE99-001/init-core-data-001
   [junit4]   2> 59310 INFO  (SUITE-ZkStateReaderTest-seed#[FF7B6F80A8BACE99]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 59313 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalCollectionWatchedNotWatched
   [junit4]   2> 59313 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 59328 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 59328 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 59424 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer start zk server on port:58545
   [junit4]   2> 59424 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 59425 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 59427 INFO  (zkCallback-41-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6312879 name:ZooKeeperConnection Watcher:127.0.0.1:58545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 59428 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 59428 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 59429 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 59432 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 59456 INFO  (zkCallback-42-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@795af37c name:ZooKeeperConnection Watcher:127.0.0.1:58545 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 59460 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 59460 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 59460 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 59469 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 59484 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 59770 INFO  (zkCallback-43-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@cdbe233 name:ZooKeeperConnection Watcher:127.0.0.1:58545/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 59770 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 59771 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 59772 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 59773 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 59775 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 59776 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 59777 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 59777 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 59778 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 59779 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 59780 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 59781 INFO  (zkCallback-43-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 59782 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 59783 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 59783 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Removing watch for uninteresting collection [c1]
   [junit4]   2> 59784 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58545 58545
   [junit4]   2> 59970 INFO  (Thread-85) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:58545 58545
   [junit4]   2> 59970 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalCollectionWatchedNotWatched
   [junit4]   2> 59972 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWatchedCollectionCreation
   [junit4]   2> 59972 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 59972 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 59972 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 60076 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer start zk server on port:37574
   [junit4]   2> 60076 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 60098 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 60114 INFO  (zkCallback-44-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@265cbc5b name:ZooKeeperConnection Watcher:127.0.0.1:37574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 60116 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60116 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 60117 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x154348022db0000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 60118 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 60119 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 60123 INFO  (zkCallback-45-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@33bad1b0 name:ZooKeeperConnection Watcher:127.0.0.1:37574 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 60123 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60123 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 60123 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 60125 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 60140 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 60152 INFO  (zkCallback-46-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@39b97145 name:ZooKeeperConnection Watcher:127.0.0.1:37574/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 60152 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60153 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 60153 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 60154 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 60155 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 60155 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 60156 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 60157 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 60157 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 60158 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 60158 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 60158 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 60159 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 60165 INFO  (zkCallback-46-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 60182 INFO  (zkCallback-46-thread-3) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 60183 INFO  (zkCallback-46-thread-3) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 60212 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37574 37574
   [junit4]   2> 60392 INFO  (Thread-86) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37574 37574
   [junit4]   2> 60396 INFO  (TEST-ZkStateReaderTest.testWatchedCollectionCreation-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWatchedCollectionCreation
   [junit4]   2> 60399 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> 60399 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 60406 INFO  (Thread-87) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 60406 INFO  (Thread-87) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 60508 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer start zk server on port:43053
   [junit4]   2> 60508 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 60532 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 60538 INFO  (zkCallback-47-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@574c3bd name:ZooKeeperConnection Watcher:127.0.0.1:43053 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 60539 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60539 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 60941 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 60941 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 60946 INFO  (zkCallback-48-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@8aeb0a9 name:ZooKeeperConnection Watcher:127.0.0.1:43053 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 60946 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60946 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 60946 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 60949 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 60956 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 60969 INFO  (zkCallback-49-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@752d5b9e name:ZooKeeperConnection Watcher:127.0.0.1:43053/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 60970 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 60970 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 60971 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 60973 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 60974 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 60974 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 60975 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 60975 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 60976 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 60976 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 60977 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 60977 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 60979 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 60992 INFO  (zkCallback-49-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 60992 WARN  (zkCallback-49-thread-1) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 61000 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$8/553631725@1bbb50ee rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@5eb9cdfd[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 2]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 61001 INFO  (zkCallback-49-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 61001 WARN  (zkCallback-49-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /clusterstate.json]
   [junit4]   2> 61002 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43053 43053
   [junit4]   2> 61422 INFO  (Thread-87) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:43053 43053
   [junit4]   2> 61462 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateReaderTest -Dtests.method=testStateFormatUpdateWithExplicitRefresh -Dtests.seed=FF7B6F80A8BACE99 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en-MT -Dtests.timezone=Pacific/Bougainville -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1.07s J0 | ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not find collection : c1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([FF7B6F80A8BACE99:E0C11E7778DA085C]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:136)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh(ZkStateReaderTest.java:42)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 61467 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 61467 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 61470 INFO  (Thread-88) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 61471 INFO  (Thread-88) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 61630 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer start zk server on port:35383
   [junit4]   2> 61631 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 61631 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61635 INFO  (zkCallback-50-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@69170e0f name:ZooKeeperConnection Watcher:127.0.0.1:35383 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 61635 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 61635 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 61637 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 61637 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61644 INFO  (zkCallback-51-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1e5c365b name:ZooKeeperConnection Watcher:127.0.0.1:35383 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 61644 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 61645 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 61645 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 61647 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 61647 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61649 INFO  (zkCallback-52-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@cef60ab name:ZooKeeperConnection Watcher:127.0.0.1:35383/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 61649 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 61650 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 61650 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 61651 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 61651 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 61652 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 61653 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 61653 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 61654 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 61654 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 61655 INFO  (zkCallback-52-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 61662 INFO  (zkCallback-52-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 61662 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 61663 INFO  (zkCallback-52-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 61729 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35383 35383
   [junit4]   2> 62189 INFO  (Thread-88) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:35383 35383
   [junit4]   2> 62190 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 62191 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> 62192 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 62204 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 62204 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 62300 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer start zk server on port:36258
   [junit4]   2> 62300 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62301 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62304 INFO  (zkCallback-53-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d5b8571 name:ZooKeeperConnection Watcher:127.0.0.1:36258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62304 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62305 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62306 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62307 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62313 INFO  (zkCallback-54-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@412a014f name:ZooKeeperConnection Watcher:127.0.0.1:36258 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62313 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62313 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62313 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 62318 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62319 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62321 INFO  (zkCallback-55-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@36adda92 name:ZooKeeperConnection Watcher:127.0.0.1:36258/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62321 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62321 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62322 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 62323 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 62324 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 62325 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 62326 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 62327 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 62328 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 62328 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 62330 INFO  (zkCallback-55-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 62330 INFO  (zkCallback-55-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 62331 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 62332 INFO  (zkCallback-55-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 62348 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36258 36258
   [junit4]   2> 62596 INFO  (Thread-89) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:36258 36258
   [junit4]   2> 62608 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> 62610 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelay
   [junit4]   2> 62610 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 62620 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 62620 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 62720 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer start zk server on port:56358
   [junit4]   2> 62720 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62736 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62772 INFO  (zkCallback-56-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@58f4391b name:ZooKeeperConnection Watcher:127.0.0.1:56358 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62773 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62773 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62787 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62793 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62797 INFO  (zkCallback-57-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4e4d5e9e name:ZooKeeperConnection Watcher:127.0.0.1:56358 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62797 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62798 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62798 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 62800 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 62800 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62803 INFO  (zkCallback-58-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4d224a27 name:ZooKeeperConnection Watcher:127.0.0.1:56358/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 62803 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62803 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 62804 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 62806 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 62807 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 62807 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 62808 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 62809 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 62809 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 62810 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 62810 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 62810 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 62816 INFO  (zkCallback-58-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 62816 INFO  (zkCallback-58-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 62862 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 62863 INFO  (zkCallback-58-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeCreated path:/collections/c1/state.json] for collection [c1] has occurred - updating... (live nodes size: [0])
   [junit4]   2> 62864 INFO  (zkCallback-58-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 62864 INFO  (zkCallback-58-thread-2) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 62866 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:56358 56358
   [junit4]   2> 63120 INFO  (Thread-90) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:56358 56358
   [junit4]   2> 63121 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelay
   [junit4]   2> 63122 INFO  (SUITE-ZkStateReaderTest-seed#[FF7B6F80A8BACE99]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_FF7B6F80A8BACE99-001
   [junit4]   2> Apr 20, 2016 4:28:16 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=en-MT, timezone=Pacific/Bougainville
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=219252240,total=329252864
   [junit4]   2> NOTE: All tests run in this JVM: [ResourceLoaderTest, TestAuthorizationFramework, OutputWriterTest, DateRangeFieldTest, TestSolrQueryParserResource, ZkStateReaderTest]
   [junit4] Completed [19/591 (1!)] on J0 in 4.39s, 6 tests, 1 error <<< FAILURES!

[...truncated 312 lines...]
   [junit4] Suite: org.apache.solr.TestDistributedSearch
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/init-core-data-001
   [junit4]   2> 87806 INFO  (SUITE-TestDistributedSearch-seed#[FF7B6F80A8BACE99]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   2> 87806 INFO  (SUITE-TestDistributedSearch-seed#[FF7B6F80A8BACE99]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /gr_y/
   [junit4]   2> 87995 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1
   [junit4]   2> 88005 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 88073 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4a2fc678{/gr_y,null,AVAILABLE}
   [junit4]   2> 88074 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.ServerConnector Started ServerConnector@54bba0e1{HTTP/1.1,[http/1.1]}{127.0.0.1:45946}
   [junit4]   2> 88074 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.Server Started @93450ms
   [junit4]   2> 88074 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/gr_y, hostPort=45946, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores}
   [junit4]   2> 88074 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 88074 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control'
   [junit4]   2> 88075 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 88075 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 88075 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/solr.xml
   [junit4]   2> 88096 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores
   [junit4]   2> 88096 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer New CoreContainer 1010270595
   [junit4]   2> 88096 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control]
   [junit4]   2> 88097 WARN  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/lib
   [junit4]   2> 88098 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 88098 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 88099 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 88099 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 88099 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 88099 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 88100 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores
   [junit4]   2> 88108 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/, configSet=collection1}
   [junit4]   2> 88108 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1
   [junit4]   2> 88108 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 88233 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 88233 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 88233 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 88233 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 88233 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 88248 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 88248 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 88282 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 88287 WARN  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 88288 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 88339 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 88356 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 88778 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 88783 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 88783 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 88812 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 88827 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 88828 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 88828 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 88830 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 88830 INFO  (coreLoadExecutor-176-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1
   [junit4]   2> 88830 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 88830 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data/]
   [junit4]   2> 88830 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@b345582
   [junit4]   2> 88830 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data
   [junit4]   2> 88831 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data/index/
   [junit4]   2> 88831 WARN  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 88831 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data/index
   [junit4]   2> 88831 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=15, maxMergeAtOnceExplicit=30, maxMergedSegmentMB=51.376953125, floorSegmentMB=0.982421875, forceMergeDeletesPctAllowed=2.314092240309382, segmentsPerTier=25.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5840385616671532
   [junit4]   2> 88849 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@75e18cca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f66c197),segFN=segments_1,generation=1}
   [junit4]   2> 88849 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 88855 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 88856 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data/
   [junit4]   2> 88856 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/control/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 88859 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 88861 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 88862 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 88862 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 88865 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 88866 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,spellCheckCompRH,lazy,/update/json/docs,/admin/luke,/export,spellCheckWithWordbreak_Direct,tvrh,/get,/admin/properties,/update/json,mltrh,/admin/threads,/search-facet-def,/update/csv,/sql,/search-facet-invariants,/admin/segments,spellCheckCompRH1,/mlt,/admin/system,/replication,dismax,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,spellCheckWithWordbreak,mock,/admin/ping,/update,/admin/file,/terms,spellCheckCompRH_Direct,/debug/dump
   [junit4]   2> 88866 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 88877 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 88877 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 88878 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1357378187, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 88878 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@75e18cca lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@2f66c197),segFN=segments_1,generation=1}
   [junit4]   2> 88878 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 88878 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@77f212db[collection1] main]
   [junit4]   2> 88879 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 88879 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 88879 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 88879 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 88902 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 88942 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 88942 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 88943 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 88943 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 88943 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 88943 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 88943 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 88943 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 88944 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 88944 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 88949 INFO  (coreLoadExecutor-176-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 88950 INFO  (searcherExecutor-177-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@77f212db[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 89473 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/cores/collection1
   [junit4]   2> 89474 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 89513 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@608f3b5d{/gr_y,null,AVAILABLE}
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.ServerConnector Started ServerConnector@13bd0324{HTTP/1.1,[http/1.1]}{127.0.0.1:42604}
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.e.j.s.Server Started @94890ms
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/gr_y, hostPort=42604, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/cores}
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0'
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 89514 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 89515 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/solr.xml
   [junit4]   2> 89524 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/cores
   [junit4]   2> 89524 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer New CoreContainer 165988372
   [junit4]   2> 89524 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0]
   [junit4]   2> 89524 WARN  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/lib
   [junit4]   2> 89524 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 89525 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 89526 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 89526 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 89526 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 89526 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 89526 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_FF7B6F80A8BACE99-001/tempDir-001/shard0/cores
   [junit4]   2> 89533 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/, configSet=collection1}
   [junit4]   2> 89533 INFO  (TEST-TestDistributedSearch.test-seed#[FF7B6F80A8BACE99]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Nightly

[...truncated too long message...]

g-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-collective-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-event-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-exception-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-journal-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-normalization-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-operational-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-referral-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-schema-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-subtree-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-interceptors-trigger-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-jdbm1-2.0.0-M2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-kerberos-codec-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-ldif-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-mavibot-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-kerberos-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-ldap-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-protocol-shared-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/apacheds-xdbm-partition-2.0.0-M15.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/api-all-1.0.0-M20.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/bcprov-jdk15-1.45.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/cglib-nodep-2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-collections-3.2.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/commons-math3-3.4.1.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/easymock-3.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/ehcache-core-2.4.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-common-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-hdfs-2.6.0-tests.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/hadoop-minikdc-2.6.0.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-annotations-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jackson-databind-2.5.4.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-core-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jersey-server-1.9.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/jetty-util-6.1.26.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mina-core-2.0.0-M5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/mockito-core-1.9.5.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/objenesis-1.2.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/presto-parser-0.122.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/test-lib/slice-0.10.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/analysis/icu/lucene-analyzers-icu-6.1.0-SNAPSHOT.jar:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/contrib/solr-analysis-extras/classes/java:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/queryparser/classes/test:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/contrib/analysis-extras/lib/icu4j-56.1.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-launcher.jar:/x1/jenkins/.ant/lib/ivy-2.3.0.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-antlr.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-regexp.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-resolver.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-net.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jmf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-oro.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-xalan2.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-commons-logging.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jsch.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit4.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-javamail.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jai.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-jdepend.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-junit.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bsf.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-swing.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-netrexx.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-bcel.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-testutil.jar:/home/jenkins/jenkins-slave/tools/hudson.tasks.Ant_AntInstallation/ant-1.8.2/lib/ant-apache-log4j.jar:/x1/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8/lib/tools.jar:/x1/jenkins/.ivy2/cache/com.carrotsearch.randomizedtesting/junit4-ant/jars/junit4-ant-2.3.4.jar com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe -eventsfile /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J0-20160420_162707_968.events @/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J0-20160420_162707_968.suites -stdin
   [junit4] ERROR: JVM J0 ended with an exception: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.executeSlave(JUnit4.java:1501)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4.access$000(JUnit4.java:121)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:955)
   [junit4] 	at com.carrotsearch.ant.tasks.junit4.JUnit4$2.call(JUnit4.java:952)
   [junit4] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4] 	at java.lang.Thread.run(Thread.java:745)

BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: At least one slave process threw an exception, first: Quit event not received from the forked process? This may indicate JVM crash or runner bugs.

Total time: 365 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 43 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/43/

2 tests failed.
FAILED:  org.apache.lucene.index.TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom

Error Message:
Captured an uncaught exception in thread: Thread[id=259, name=Thread-179, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=259, name=Thread-179, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
	at java.util.HashMap.newNode(HashMap.java:1734)
	at java.util.HashMap.putVal(HashMap.java:630)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:689)
	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:678)
	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:100)
	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256)
	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:130)
	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:220)
	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.TestIndexWriterThreadsToSegments

Error Message:
The test or suite printed 18030 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true

Stack Trace:
java.lang.AssertionError: The test or suite printed 18030 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
	at org.apache.lucene.util.TestRuleLimitSysouts.afterIfSuccessful(TestRuleLimitSysouts.java:211)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 690 lines...]
   [junit4] Suite: org.apache.lucene.index.TestIndexWriterThreadsToSegments
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-179,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at java.util.HashMap.newNode(HashMap.java:1734)
   [junit4]   2> 	at java.util.HashMap.putVal(HashMap.java:630)
   [junit4]   2> 	at java.util.HashMap.put(HashMap.java:611)
   [junit4]   2> 	at java.util.HashSet.add(HashSet.java:219)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:689)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:678)
   [junit4]   2> 	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:100)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
   [junit4]   2> 	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291)
   [junit4]   2> 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
   [junit4]   2> 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256)
   [junit4]   2> 	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:130)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:220)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-184,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-181,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-183,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-180,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-178,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-177,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-182,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-176,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> Nis 20, 2016 1:55:55 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-185,5,TGRP-TestIndexWriterThreadsToSegments]
   [junit4]   2> java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]   2> Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]   2> 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]   2> 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterThreadsToSegments -Dtests.method=testSegmentCountOnFlushRandom -Dtests.seed=11A92BEEB7FC60E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Asia/Dacca -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   1143s J0 | TestIndexWriterThreadsToSegments.testSegmentCountOnFlushRandom <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=259, name=Thread-179, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at java.util.HashMap.newNode(HashMap.java:1734)
   [junit4]    > 	at java.util.HashMap.putVal(HashMap.java:630)
   [junit4]    > 	at java.util.HashMap.put(HashMap.java:611)
   [junit4]    > 	at java.util.HashSet.add(HashSet.java:219)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter$ReaderPool.noDups(IndexWriter.java:689)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:678)
   [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:100)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:444)
   [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:291)
   [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:266)
   [junit4]    > 	at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:256)
   [junit4]    > 	at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$CheckSegmentCount.run(TestIndexWriterThreadsToSegments.java:130)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:220)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=265, name=Thread-185, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #3: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=256, name=Thread-176, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=262, name=Thread-182, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=257, name=Thread-177, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=258, name=Thread-178, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #7: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=260, name=Thread-180, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=263, name=Thread-183, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=261, name=Thread-181, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=264, name=Thread-184, state=RUNNABLE, group=TGRP-TestIndexWriterThreadsToSegments]
   [junit4]    > Caused by: java.lang.RuntimeException: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:218)
   [junit4]    > Caused by: java.util.concurrent.BrokenBarrierException
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.dowait(CyclicBarrier.java:250)
   [junit4]    > 	at java.util.concurrent.CyclicBarrier.await(CyclicBarrier.java:362)
   [junit4]    > 	at org.apache.lucene.index.TestIndexWriterThreadsToSegments$2.run(TestIndexWriterThreadsToSegments.java:215)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {field=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, maxPointsInLeafNode=469, maxMBSortInHeap=6.249063691917417, sim=RandomSimilarity(queryNorm=false,coord=yes): {field=IB SPL-D3(800.0)}, locale=tr, timezone=Asia/Dacca
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=284180056,total=355467264
   [junit4]   2> NOTE: All tests run in this JVM: [TestNoMergePolicy, TestSortedNumericSortField, TestCheckIndex, TestTerm, TestDeterminizeLexicon, TestSloppyPhraseQuery2, TestPersistentSnapshotDeletionPolicy, TestTermVectorsWriter, TestMergeRateLimiter, TestDocCount, TestMmapDirectory, TestParallelCompositeReader, TestSegmentInfos, TestDirectMonotonic, TestTransactionRollback, TestSortRandom, TestIsCurrent, TestFixedBitDocIdSet, TestCustomNorms, TestNRTReaderCleanup, TestPriorityQueue, TestLockFactory, TestSimilarityProvider, TestIndexWriterDelete, TestHighCompressionMode, TestSwappedIndexFiles, TestSimpleAttributeImpl, TestForTooMuchCloning, TestDeterminism, TestBlockPostingsFormat2, TestIndexWriterThreadsToSegments]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestIndexWriterThreadsToSegments -Dtests.seed=11A92BEEB7FC60E -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Asia/Dacca -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | TestIndexWriterThreadsToSegments (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: The test or suite printed 18030 bytes to stdout and stderr, even though the limit was set to 8192 bytes. Increase the limit with @Limit, ignore it completely with @SuppressSysoutChecks or run with -Dtests.verbose=true
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([11A92BEEB7FC60E]:0)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [143/419 (1!)] on J0 in 1146.01s, 4 tests, 1 failure, 1 error <<< FAILURES!

[...truncated 979 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/test/temp/junit4-J0-20160419_193335_073.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: GC overhead limit exceeded
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid31843.hprof ...
   [junit4] Heap dump file created [534641087 bytes in 0.957 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 18 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 419 suites (1 ignored), 3502 tests, 1 suite-level error, 1 error, 40 ignored (36 assumptions) [seed: 11A92BEEB7FC60E]

Total time: 77 minutes 33 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




Re: [JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 42 - Still Failing

Posted by Robert Muir <rc...@gmail.com>.
NOTE: test params are:
codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=1, maxDocsPerChunk=772, blockSize=433),
termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION,
chunkSize=1, blockSize=433))

This seems like an extremely slow case for what this test is doing.
The format being used is more theoretical and parameters are geared at
allowing smaller tests to find bugs, not at huge monsterish tests like
this one.

I will try to (carefully) tone down the test.

On Mon, Apr 18, 2016 at 7:15 AM, Apache Jenkins Server
<je...@builds.apache.org> wrote:
> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/42/
>
> 2 tests failed.
> FAILED:  org.apache.lucene.index.Test4GBStoredFields.test
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
>         at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)
>
>
> FAILED:  junit.framework.TestSuite.org.apache.lucene.index.Test4GBStoredFields
>
> Error Message:
> Suite timeout exceeded (>= 14400000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 14400000 msec).
>         at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)
>
>
>
>
> Build Log:
> [...truncated 1897 lines...]
>    [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
>    [junit4]   2> 四月 18, 2016 7:14:48 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
>    [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.Test4GBStoredFields
>    [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
>    [junit4]   2>         at java.lang.Object.wait(Native Method)
>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1245)
>    [junit4]   2>         at java.lang.Thread.join(Thread.java:1319)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:601)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)
>    [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:243)
>    [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:354)
>    [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
>    [junit4]   2>    2) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
>    [junit4]   2>         at java.lang.Thread.sleep(Native Method)
>    [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
>    [junit4]   2>    3) Thread[id=5437, name=TEST-Test4GBStoredFields.test-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.LZ4.compressHC(LZ4.java:423)
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressionMode$LZ4HighCompressor.compress(CompressionMode.java:180)
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:233)
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:163)
>    [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:343)
>    [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:384)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)
>    [junit4]   2>         at org.apache.lucene.index.Test4GBStoredFields.test(Test4GBStoredFields.java:75)
>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2>    4) Thread[id=5436, name=SUITE-Test4GBStoredFields-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
>    [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1552)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:688)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:685)
>    [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:685)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:701)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:414)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:681)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test4GBStoredFields -Dtests.method=test -Dtests.seed=68168265E2F0BC79 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh-TW -Dtests.timezone=America/Guyana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>    [junit4] ERROR   14400s J1 | Test4GBStoredFields.test <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)
>    [junit4]   2> 四月 18, 2016 7:14:49 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
>    [junit4]   2> 四月 18, 2016 7:15:09 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
>    [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.Test4GBStoredFields:
>    [junit4]   2>    1) Thread[id=5437, name=TEST-Test4GBStoredFields.test-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.LZ4.compressHC(LZ4.java:423)
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressionMode$LZ4HighCompressor.compress(CompressionMode.java:180)
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:233)
>    [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:163)
>    [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:343)
>    [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:384)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
>    [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)
>    [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)
>    [junit4]   2>         at org.apache.lucene.index.Test4GBStoredFields.test(Test4GBStoredFields.java:75)
>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>    [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>    [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>    [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
>    [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
>    [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
>    [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
>    [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
>    [junit4]   2> 四月 18, 2016 7:15:09 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> INFO: Starting to interrupt leaked threads:
>    [junit4]   2>    1) Thread[id=5437, name=TEST-Test4GBStoredFields.test-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
>    [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/test/J1/temp/lucene.index.Test4GBStoredFields_68168265E2F0BC79-001
>    [junit4]   2> Apr 18, 2016 11:15:09 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
>    [junit4]   2> INFO: All leaked threads terminated.
>    [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=1, maxDocsPerChunk=772, blockSize=433), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=1, blockSize=433)), sim=ClassicSimilarity, locale=zh-TW, timezone=America/Guyana
>    [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=75486216,total=405798912
>    [junit4]   2> NOTE: All tests run in this JVM: [TestStressIndexing, TestDocsAndPositions, TestIndexWriterExceptions2, TestPostingsOffsets, TestBoolean2, TestTryDelete, TestLucene50FieldInfoFormat, Test2BPoints, TestControlledRealTimeReopenThread, TestPayloads, TestRollingBuffer, TestNRTCachingDirectory, TestSortedSetSelector, TestPositionIncrement, TestNotDocIdSet, TestPerFieldDocValuesFormat, TestStringHelper, TestCollectionUtil, TestSimpleExplanationsOfNonMatches, TestQueryBuilder, TestRAMDirectory, TestTopFieldCollector, TestNIOFSDirectory, TestStressAdvance, TestSimpleFSDirectory, TestMultiPhraseEnum, TestCompiledAutomaton, TestSpanSearchEquivalence, TestBagOfPostings, TestTermdocPerf, TestParallelLeafReader, TestDirectoryReaderReopen, TestApproximationSearchEquivalence, TestIntArrayDocIdSet, TestSnapshotDeletionPolicy, TestOmitPositions, TestBooleanQuery, TestPackedInts, TestFieldMaskingSpanQuery, TestAutomatonQueryUnicode, TestPerFieldPostingsFormat2, TestNumericDocValuesUpdates, TestFileSwitchDirectory, TestFilterSpans, TestSegmentTermDocs, TestReqExclBulkScorer, TestBasics, TestBufferedChecksum, Test2BPostingsBytes, TestGeoEncodingUtils, TestFixedLengthBytesRefArray, TestWildcardRandom, MultiCollectorTest, TestSetOnce, TestSimilarity, TestPrefixRandom, TestLucene54DocValuesFormat, TestRegexpQuery, TestOmitTf, TestGrowableByteArrayDataOutput, TestRollback, TestSleepingLockWrapper, TestFlex, TestSpanOrQuery, TestSearchAfter, TestSegmentMerger, TestIndexWriterOnJRECrash, TestAssertions, TestMatchAllDocsQuery, TestMatchNoDocsQuery, TestToken, TestFieldsReader, TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, TestBooleanMinShouldMatch, TestTermsEnum2, TestLegacyNumericUtils, TestDeterminizeLexicon, TestIndexWriterOnDiskFull, TestTopDocsCollector, TestUniqueTermCount, TestDocument, TestTermScorer, TestSumDocFreq, TestDocValuesScoring, TestDirectory, TestPayloadsOnVectors, TestSearchForDuplicates, TestParallelTermEnum, TestElevationComparator, TestRecyclingIntBlockAllocator, TestMultiTermQueryRewrites, TestDateSort, TestSpanExplanationsOfNonMatches, TestVirtualMethod, TestNumericTokenStream, TestSearch, TestPriorityQueue, TestSameTokenSamePosition, TestDocCount, TestCachingCollector, TestTotalHitCountCollector, TestRecyclingByteBlockAllocator, TestBooleanScorer, TestRamUsageEstimator, TestNoMergePolicy, TestNoMergeScheduler, TestNamedSPILoader, TestIndexWriterLockRelease, TestByteBlockPool, TestDemo, TestTwoPhaseCommitTool, TestCloseableThreadLocal, TestVersion, Test4GBStoredFields]
>    [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
>    [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test4GBStoredFields -Dtests.seed=68168265E2F0BC79 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh-TW -Dtests.timezone=America/Guyana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>    [junit4] ERROR   0.00s J1 | Test4GBStoredFields (suite) <<<
>    [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 14400000 msec).
>    [junit4]    >        at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)
>    [junit4] Completed [419/419 (1!)] on J1 in 14420.13s, 1 test, 2 errors <<< FAILURES!
>
> [...truncated 12 lines...]
> BUILD FAILED
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build.xml:50: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
> /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 419 suites (1 ignored), 3501 tests, 1 suite-level error, 1 error, 39 ignored (35 assumptions) [seed: 68168265E2F0BC79]
>
> Total time: 289 minutes 40 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> No prior successful build to compare, so performing full copy of artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 42 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/42/

2 tests failed.
FAILED:  org.apache.lucene.index.Test4GBStoredFields.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.index.Test4GBStoredFields

Error Message:
Suite timeout exceeded (>= 14400000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 14400000 msec).
	at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)




Build Log:
[...truncated 1897 lines...]
   [junit4] Suite: org.apache.lucene.index.Test4GBStoredFields
   [junit4]   2> 四月 18, 2016 7:14:48 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.index.Test4GBStoredFields
   [junit4]   2>    1) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1245)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1319)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:601)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:243)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:354)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
   [junit4]   2>    2) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2>    3) Thread[id=5437, name=TEST-Test4GBStoredFields.test-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
   [junit4]   2>         at org.apache.lucene.codecs.compressing.LZ4.compressHC(LZ4.java:423)
   [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressionMode$LZ4HighCompressor.compress(CompressionMode.java:180)
   [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:233)
   [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:163)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:343)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:384)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)
   [junit4]   2>         at org.apache.lucene.index.Test4GBStoredFields.test(Test4GBStoredFields.java:75)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    4) Thread[id=5436, name=SUITE-Test4GBStoredFields-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:688)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:685)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:685)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:701)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:414)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test4GBStoredFields -Dtests.method=test -Dtests.seed=68168265E2F0BC79 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh-TW -Dtests.timezone=America/Guyana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   14400s J1 | Test4GBStoredFields.test <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)
   [junit4]   2> 四月 18, 2016 7:14:49 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> 四月 18, 2016 7:15:09 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.index.Test4GBStoredFields: 
   [junit4]   2>    1) Thread[id=5437, name=TEST-Test4GBStoredFields.test-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
   [junit4]   2>         at org.apache.lucene.codecs.compressing.LZ4.compressHC(LZ4.java:423)
   [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressionMode$LZ4HighCompressor.compress(CompressionMode.java:180)
   [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.flush(CompressingStoredFieldsWriter.java:233)
   [junit4]   2>         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.finishDocument(CompressingStoredFieldsWriter.java:163)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.finishStoredFields(DefaultIndexingChain.java:343)
   [junit4]   2>         at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:384)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:232)
   [junit4]   2>         at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:449)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1492)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1271)
   [junit4]   2>         at org.apache.lucene.index.Test4GBStoredFields.test(Test4GBStoredFields.java:75)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 四月 18, 2016 7:15:09 上午 com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=5437, name=TEST-Test4GBStoredFields.test-seed#[68168265E2F0BC79], state=RUNNABLE, group=TGRP-Test4GBStoredFields]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/test/J1/temp/lucene.index.Test4GBStoredFields_68168265E2F0BC79-001
   [junit4]   2> Apr 18, 2016 11:15:09 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=1, maxDocsPerChunk=772, blockSize=433), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=1, blockSize=433)), sim=ClassicSimilarity, locale=zh-TW, timezone=America/Guyana
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=75486216,total=405798912
   [junit4]   2> NOTE: All tests run in this JVM: [TestStressIndexing, TestDocsAndPositions, TestIndexWriterExceptions2, TestPostingsOffsets, TestBoolean2, TestTryDelete, TestLucene50FieldInfoFormat, Test2BPoints, TestControlledRealTimeReopenThread, TestPayloads, TestRollingBuffer, TestNRTCachingDirectory, TestSortedSetSelector, TestPositionIncrement, TestNotDocIdSet, TestPerFieldDocValuesFormat, TestStringHelper, TestCollectionUtil, TestSimpleExplanationsOfNonMatches, TestQueryBuilder, TestRAMDirectory, TestTopFieldCollector, TestNIOFSDirectory, TestStressAdvance, TestSimpleFSDirectory, TestMultiPhraseEnum, TestCompiledAutomaton, TestSpanSearchEquivalence, TestBagOfPostings, TestTermdocPerf, TestParallelLeafReader, TestDirectoryReaderReopen, TestApproximationSearchEquivalence, TestIntArrayDocIdSet, TestSnapshotDeletionPolicy, TestOmitPositions, TestBooleanQuery, TestPackedInts, TestFieldMaskingSpanQuery, TestAutomatonQueryUnicode, TestPerFieldPostingsFormat2, TestNumericDocValuesUpdates, TestFileSwitchDirectory, TestFilterSpans, TestSegmentTermDocs, TestReqExclBulkScorer, TestBasics, TestBufferedChecksum, Test2BPostingsBytes, TestGeoEncodingUtils, TestFixedLengthBytesRefArray, TestWildcardRandom, MultiCollectorTest, TestSetOnce, TestSimilarity, TestPrefixRandom, TestLucene54DocValuesFormat, TestRegexpQuery, TestOmitTf, TestGrowableByteArrayDataOutput, TestRollback, TestSleepingLockWrapper, TestFlex, TestSpanOrQuery, TestSearchAfter, TestSegmentMerger, TestIndexWriterOnJRECrash, TestAssertions, TestMatchAllDocsQuery, TestMatchNoDocsQuery, TestToken, TestFieldsReader, TestDocValuesIndexing, TestSimilarityBase, TestHugeRamFile, TestBooleanMinShouldMatch, TestTermsEnum2, TestLegacyNumericUtils, TestDeterminizeLexicon, TestIndexWriterOnDiskFull, TestTopDocsCollector, TestUniqueTermCount, TestDocument, TestTermScorer, TestSumDocFreq, TestDocValuesScoring, TestDirectory, TestPayloadsOnVectors, TestSearchForDuplicates, TestParallelTermEnum, TestElevationComparator, TestRecyclingIntBlockAllocator, TestMultiTermQueryRewrites, TestDateSort, TestSpanExplanationsOfNonMatches, TestVirtualMethod, TestNumericTokenStream, TestSearch, TestPriorityQueue, TestSameTokenSamePosition, TestDocCount, TestCachingCollector, TestTotalHitCountCollector, TestRecyclingByteBlockAllocator, TestBooleanScorer, TestRamUsageEstimator, TestNoMergePolicy, TestNoMergeScheduler, TestNamedSPILoader, TestIndexWriterLockRelease, TestByteBlockPool, TestDemo, TestTwoPhaseCommitTool, TestCloseableThreadLocal, TestVersion, Test4GBStoredFields]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=Test4GBStoredFields -Dtests.seed=68168265E2F0BC79 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=zh-TW -Dtests.timezone=America/Guyana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | Test4GBStoredFields (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 14400000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([68168265E2F0BC79]:0)
   [junit4] Completed [419/419 (1!)] on J1 in 14420.13s, 1 test, 2 errors <<< FAILURES!

[...truncated 12 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 419 suites (1 ignored), 3501 tests, 1 suite-level error, 1 error, 39 ignored (35 assumptions) [seed: 68168265E2F0BC79]

Total time: 289 minutes 40 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 41 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/41/

4 tests failed.
FAILED:  org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92:85C280C4570CEA0C]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.ZkSolrClientTest.testMultipleWatchesAsync(ZkSolrClientTest.java:265)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest.test

Error Message:
There are still nodes recoverying - waited for 440 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 440 seconds
	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92:65206CBC6C14866A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:858)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1414)
	at org.apache.solr.cloud.TlogReplayBufferedWhileIndexingTest.test(TlogReplayBufferedWhileIndexingTest.java:99)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=61554, name=collection3, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=61554, name=collection3, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40663: collection already exists: awholynewstresscollection_collection3_9
	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=51927, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=51927, name=collection2, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:49462/n_x: collection already exists: awholynewstresscollection_collection2_0
	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)




Build Log:
[...truncated 11415 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/init-core-data-001
   [junit4]   2> 233691 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /whsb/
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 233760 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 233788 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 233790 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 233823 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_58090_hdfs____rj3mge/webapp
   [junit4]   2> 234119 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 234872 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:58090
   [junit4]   2> 235215 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 235216 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 235238 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_60935_datanode____juqsi0/webapp
   [junit4]   2> 235506 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 235885 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60935
   [junit4]   2> 236385 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 236386 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 236452 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_40882_datanode____.622wv7/webapp
   [junit4]   2> 236679 INFO  (IPC Server handler 6 on 46223) [    ] BlockStateChange BLOCK* processReport: from storage DS-6929095b-6bcb-43bd-b729-e299a35b4efd node DatanodeRegistration(127.0.0.1, datanodeUuid=5d74ff2b-521b-4414-8eab-dcc974579e14, infoPort=60935, ipcPort=37983, storageInfo=lv=-56;cid=testClusterID;nsid=1358635743;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 236680 INFO  (IPC Server handler 6 on 46223) [    ] BlockStateChange BLOCK* processReport: from storage DS-0a2be1e2-2a1e-4f3f-996e-5044013f40e0 node DatanodeRegistration(127.0.0.1, datanodeUuid=5d74ff2b-521b-4414-8eab-dcc974579e14, infoPort=60935, ipcPort=37983, storageInfo=lv=-56;cid=testClusterID;nsid=1358635743;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 236688 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 237002 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[ED745366C2E8EB92]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40882
   [junit4]   2> 237294 INFO  (IPC Server handler 8 on 46223) [    ] BlockStateChange BLOCK* processReport: from storage DS-2fc05e9e-1b17-4447-a6db-d03b51f67154 node DatanodeRegistration(127.0.0.1, datanodeUuid=f8e3df72-5fe6-4372-88ad-572cce3d4ee2, infoPort=40882, ipcPort=52600, storageInfo=lv=-56;cid=testClusterID;nsid=1358635743;c=0), blocks: 0, hasStaleStorages: true, processing time: 2 msecs
   [junit4]   2> 237294 INFO  (IPC Server handler 8 on 46223) [    ] BlockStateChange BLOCK* processReport: from storage DS-cfb0d7e7-af1e-487f-90fe-c6287124a665 node DatanodeRegistration(127.0.0.1, datanodeUuid=f8e3df72-5fe6-4372-88ad-572cce3d4ee2, infoPort=40882, ipcPort=52600, storageInfo=lv=-56;cid=testClusterID;nsid=1358635743;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 237576 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 237586 INFO  (Thread-5487) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 237586 INFO  (Thread-5487) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 237686 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ZkTestServer start zk server on port:59581
   [junit4]   2> 237686 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 237702 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 237734 INFO  (zkCallback-90-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6d7756d4 name:ZooKeeperConnection Watcher:127.0.0.1:59581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 237738 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 237738 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 237738 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 237786 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 237786 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 237792 INFO  (zkCallback-91-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28f837c1 name:ZooKeeperConnection Watcher:127.0.0.1:59581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 237792 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 237792 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 237792 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 237793 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 237794 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 237795 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 237807 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 237807 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 237815 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 237815 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 237816 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 237816 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 237817 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 237817 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 237818 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 237818 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 237819 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 237819 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 237820 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 237820 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 237821 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 237821 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 237822 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 237822 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 237822 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 237823 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 237823 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 237823 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 237887 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores/collection1
   [junit4]   2> 237888 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 237898 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@20394e10{/whsb,null,AVAILABLE}
   [junit4]   2> 237898 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.ServerConnector Started ServerConnector@36b55a76{HTTP/1.1,[http/1.1]}{127.0.0.1:54363}
   [junit4]   2> 237898 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.Server Started @242441ms
   [junit4]   2> 237898 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:46223/hdfs__localhost_46223__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001_tempDir-002_control_data, hostContext=/whsb, hostPort=54363, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores}
   [junit4]   2> 237899 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 237899 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001'
   [junit4]   2> 237899 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 237899 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 237899 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 237913 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 238064 INFO  (zkCallback-92-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@18af2adc name:ZooKeeperConnection Watcher:127.0.0.1:59581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 238064 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 238064 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 238065 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 238066 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/solr.xml
   [junit4]   2> 238070 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores
   [junit4]   2> 238071 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CoreContainer New CoreContainer 171810031
   [junit4]   2> 238071 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001]
   [junit4]   2> 238071 WARN  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/lib
   [junit4]   2> 238071 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 238073 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 238073 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 238073 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 238073 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59581/solr
   [junit4]   2> 238073 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 238073 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 238077 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 238106 INFO  (zkCallback-95-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5a8ce7d8 name:ZooKeeperConnection Watcher:127.0.0.1:59581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 238114 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 238114 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 238124 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$17/99522583@4cc3c83e
   [junit4]   2> 238142 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 238163 INFO  (zkCallback-96-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@644150ae name:ZooKeeperConnection Watcher:127.0.0.1:59581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 238166 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 238178 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 238222 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 238274 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 238322 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 238391 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 238391 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 238392 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 238393 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 238393 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 238394 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 238394 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 238395 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 238396 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 238396 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 238396 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95742033581768708-127.0.0.1:54363_whsb-n_0000000000
   [junit4]   2> 238397 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54363_whsb
   [junit4]   2> 238397 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 238406 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer Overseer (id=95742033581768708-127.0.0.1:54363_whsb-n_0000000000) starting
   [junit4]   2> 238407 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 238409 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 238422 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 238422 INFO  (OverseerCollectionConfigSetProcessor-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 238426 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54363_whsb
   [junit4]   2> 238426 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:54363_whsb
   [junit4]   2> 238450 INFO  (zkCallback-96-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 238451 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 238451 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 238452 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 238453 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores
   [junit4]   2> 238453 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 238454 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores/collection1
   [junit4]   2> 238454 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 238454 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 238454 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 238456 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 238457 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 238457 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [n:127.0.0.1:54363_whsb    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 238457 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54363/whsb",
   [junit4]   2>   "node_name":"127.0.0.1:54363_whsb",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 238457 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54363/whsb",
   [junit4]   2>   "node_name":"127.0.0.1:54363_whsb",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 238457 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 238458 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 238458 INFO  (zkCallback-96-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 239613 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 239613 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 239614 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 239614 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 239614 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 239614 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores/collection1'
   [junit4]   2> 239614 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 239614 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 239615 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 239626 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 239629 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 239644 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 239653 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 239790 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 239790 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 239791 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 239811 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:46223/solr_hdfs_home
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/control-001/cores/collection1], dataDir=[hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 239854 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3cd135fc
   [junit4]   2> 239855 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 240059 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 240071 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 240077 WARN  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:46223/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 240077 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 240085 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 240085 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 240085 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 240152 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 240152 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 240152 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=47, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2622464418959967]
   [junit4]   2> 240414 INFO  (IPC Server handler 5 on 46223) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:56821 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2fc05e9e-1b17-4447-a6db-d03b51f67154:NORMAL:127.0.0.1:56821|RBW], ReplicaUnderConstruction[[DISK]DS-0a2be1e2-2a1e-4f3f-996e-5044013f40e0:NORMAL:127.0.0.1:35966|RBW]]} size 71
   [junit4]   2> 240414 INFO  (IPC Server handler 7 on 46223) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35966 is added to blk_1073741825_1001 size 71
   [junit4]   2> 240822 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7950687c); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 240822 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 240844 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 240844 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 240844 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 240844 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 240844 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 240844 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 240845 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 240845 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 240845 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 240845 WARN  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 240846 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 240847 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 240848 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 240848 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 240858 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 240861 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 240861 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 240872 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 240872 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 240872 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 240926 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 10000ms; 
   [junit4]   2> 240926 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 3000ms; 
   [junit4]   2> 240942 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=34, maxMergeAtOnceExplicit=13, maxMergedSegmentMB=83.0283203125, floorSegmentMB=1.6005859375, forceMergeDeletesPctAllowed=21.58651266998546, segmentsPerTier=2.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.5682205468789964
   [junit4]   2> 241022 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:46223/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@7950687c); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 241022 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 241022 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@214bec03[collection1] main]
   [junit4]   2> 241022 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 241030 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 241030 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 241032 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 241032 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 241032 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 241034 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 241034 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 241034 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 241034 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 241035 INFO  (searcherExecutor-246-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@214bec03[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 241038 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 241038 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 241038 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 241038 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1531872540919595008
   [junit4]   2> 241039 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1531872540919595008
   [junit4]   2> 241039 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 241042 INFO  (coreLoadExecutor-245-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 241050 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:54363/whsb collection:control_collection shard:shard1
   [junit4]   2> 241058 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 241086 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95742033581768708-core_node1-n_0000000000
   [junit4]   2> 241090 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=300000
   [junit4]   2> 241102 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 241102 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 241103 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 241103 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54363/whsb/collection1/
   [junit4]   2> 241103 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 241103 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:54363/whsb/collection1/ has no replicas
   [junit4]   2> 241106 INFO  (zkCallback-96-thread-1-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 241110 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 241111 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95742033581768708-core_node1-n_0000000000
   [junit4]   2> 241113 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54363/whsb/collection1/ shard1
   [junit4]   2> 241115 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 241115 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 241115 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:54363/whsb",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 241130 INFO  (zkCallback-98-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@76ca5f34 name:ZooKeeperConnection Watcher:127.0.0.1:59581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 241130 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 241130 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 241130 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 241132 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 241132 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 241220 INFO  (zkCallback-96-thread-2-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 241221 INFO  (zkCallback-98-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 241232 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001/cores/collection1
   [junit4]   2> 241233 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001
   [junit4]   2> 241246 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 241274 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:54363/whsb/collection1/ and leader is http://127.0.0.1:54363/whsb/collection1/
   [junit4]   2> 241274 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:54363/whsb
   [junit4]   2> 241274 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 241274 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 241274 INFO  (coreZkRegister-239-thread-1-processing-n:127.0.0.1:54363_whsb x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:54363_whsb c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 241275 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54363/whsb",
   [junit4]   2>   "node_name":"127.0.0.1:54363_whsb",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 3
   [junit4]   2> 241275 INFO  (OverseerStateUpdate-95742033581768708-127.0.0.1:54363_whsb-n_0000000000) [n:127.0.0.1:54363_whsb    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:54363/whsb",
   [junit4]   2>   "node_name":"127.0.0.1:54363_whsb",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 241370 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@faeb6fe{/whsb,null,AVAILABLE}
   [junit4]   2> 241370 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.ServerConnector Started ServerConnector@647d8068{HTTP/1.1,[http/1.1]}{127.0.0.1:34666}
   [junit4]   2> 241370 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.e.j.s.Server Started @245913ms
   [junit4]   2> 241370 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:46223/hdfs__localhost_46223__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001_tempDir-002_jetty1, solrconfig=solrconfig.xml, hostContext=/whsb, hostPort=34666, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001/cores}
   [junit4]   2> 241371 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 241371 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001'
   [junit4]   2> 241371 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 241371 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 241371 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 241377 INFO  (zkCallback-96-thread-2-processing-n:127.0.0.1:54363_whsb) [n:127.0.0.1:54363_whsb    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 241386 INFO  (zkCallback-98-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 241642 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 241874 INFO  (zkCallback-99-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@14dd3f57 name:ZooKeeperConnection Watcher:127.0.0.1:59581/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 241874 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 241874 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 241891 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 241892 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001/solr.xml
   [junit4]   2> 241896 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001/cores
   [junit4]   2> 241896 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CoreContainer New CoreContainer 1760517915
   [junit4]   2> 241896 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001]
   [junit4]   2> 241896 WARN  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_ED745366C2E8EB92-001/shard-1-001/lib
   [junit4]   2> 241931 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 241932 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 241932 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 241932 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 241932 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:59581/solr
   [junit4]   2> 241932 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 241932 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 241938 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 241958 INFO  (zkCallback-102-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3e14e56a name:ZooKeeperConnection Watcher:127.0.0.1:59581 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 241970 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 241970 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[ED745366C2E8EB92]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 241990 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexi

[...truncated too long message...]

bZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
   [junit4]    > 	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)Throwable #4: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=61552, name=collection1, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40663: Could not find collection : awholynewstresscollection_collection1_8
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:977)Throwable #5: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=61556, name=collection5, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40663: Could not find collection : awholynewstresscollection_collection5_9
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:977)Throwable #6: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=61553, name=collection2, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40663: Could not find collection : awholynewstresscollection_collection2_9
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([ED745366C2E8EB92]:0)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
   [junit4]    > 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
   [junit4]    > 	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:977)
   [junit4]   2> 3737436 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[ED745366C2E8EB92]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Apr 17, 2016 4:36:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=6, maxDocsPerChunk=9, blockSize=112), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=6, blockSize=112)), sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=id, timezone=Asia/Jayapura
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=181038400,total=508035072
   [junit4]   2> NOTE: All tests run in this JVM: [TestFastWriter, TestFieldTypeResource, TestCollectionAPI, TestTestInjection, TestFreeTextSuggestions, LukeRequestHandlerTest, TestSearcherReuse, BlockDirectoryTest, BitVectorTest, AutoCommitTest, TestFuzzyAnalyzedSuggestions, XmlUpdateRequestHandlerTest, SearchHandlerTest, CoreAdminCreateDiscoverTest, OverseerCollectionConfigSetProcessorTest, TestExceedMaxTermLength, ReplicationFactorTest, TestLRUStatsCache, CoreAdminHandlerTest, AssignTest, ConnectionManagerTest, SuggestComponentContextFilterQueryTest, DistributedFacetPivotWhiteBoxTest, UpdateParamsTest, DistributedDebugComponentTest, TestCollationFieldDocValues, HdfsWriteToMultipleCollectionsTest, TestReversedWildcardFilterFactory, TestElisionMultitermQuery, CurrencyFieldXmlFileTest, CloudMLTQParserTest, TestCSVResponseWriter, RegexBoostProcessorTest, BlockJoinFacetRandomTest, CleanupOldIndexTest, SolrCoreCheckLockOnStartupTest, HLLSerializationTest, SimpleFacetsTest, EnumFieldTest, HdfsBasicDistributedZkTest, TestSearchPerf, DeleteLastCustomShardedReplicaTest, TestLeaderInitiatedRecoveryThread, InfoHandlerTest, TestDistribIDF, SuggesterWFSTTest, SampleTest, HdfsCollectionsAPIDistributedZkTest, NoCacheHeaderTest, HighlighterConfigTest, TestCustomDocTransformer, DirectUpdateHandlerOptimizeTest, TestRandomRequestDistribution, TestPerFieldSimilarityWithDefaultOverride, TestAuthorizationFramework, TestCloudManagedSchema, CloneFieldUpdateProcessorFactoryTest, JavabinLoaderTest, HdfsThreadLeakTest, TestSolrQueryParserDefaultOperatorResource, TestSolrJ, TestSweetSpotSimilarityFactory, TestSolrIndexConfig, FieldMutatingUpdateProcessorTest, TestAnalyzeInfixSuggestions, TestConfigSetsAPIZkFailure, TestUseDocValuesAsStored, TestTolerantSearch, FullSolrCloudDistribCmdsTest, CdcrRequestHandlerTest, MultiThreadedOCPTest, TestExtendedDismaxParser, RecoveryZkTest, DistribJoinFromCollectionTest, TestIndexingPerformance, CollectionStateFormat2Test, PluginInfoTest, UnloadDistributedZkTest, TestStressUserVersions, TestDFISimilarityFactory, TestJsonFacets, TestMissingGroups, TestIntervalFaceting, TestManagedStopFilterFactory, BadComponentTest, TermVectorComponentDistributedTest, TestSolrDeletionPolicy1, AddSchemaFieldsUpdateProcessorFactoryTest, ForceLeaderTest, RuleEngineTest, RequestHandlersTest, TestReplicationHandlerBackup, ClusterStateTest, SuggesterTSTTest, SharedFSAutoReplicaFailoverTest, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
   [junit4] Completed [275/587 (4!)] on J2 in 477.78s, 1 test, 1 error <<< FAILURES!

[...truncated 1020 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 587 suites (10 ignored), 2366 tests, 2 errors, 2 failures, 31 ignored (16 assumptions) [seed: ED745366C2E8EB92]

Total time: 332 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 40 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/40/

4 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior

Error Message:
Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={   "shards":{"shard1":{       "parent":null,       "range":null,       "state":"active",       "replicas":{"core_node1":{           "base_url":"http://127.0.0.1/solr",           "node_name":"node1",           "core":"core1",           "roles":"",           "state":"down"}}}},   "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}

Stack Trace:
java.lang.AssertionError: Illegal state, was: down expected:active clusterState:live nodes:[]collections:{c1=DocCollection(c1)={
  "shards":{"shard1":{
      "parent":null,
      "range":null,
      "state":"active",
      "replicas":{"core_node1":{
          "base_url":"http://127.0.0.1/solr",
          "node_name":"node1",
          "core":"core1",
          "roles":"",
          "state":"down"}}}},
  "router":{"name":"implicit"}}, test=LazyCollectionRef(test)}
	at __randomizedtesting.SeedInfo.seed([6D29FDC01C68A451:537FE2CFEF8FE1F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:234)
	at org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1271)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.store.hdfs.HdfsDirectoryTest

Error Message:
Captured an uncaught exception in thread: Thread[id=14245, name=Thread-8241, state=RUNNABLE, group=TGRP-HdfsDirectoryTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=14245, name=Thread-8241, state=RUNNABLE, group=TGRP-HdfsDirectoryTest]
Caused by: java.lang.IllegalStateException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data1/current/BP-343369792-10.41.0.5-1460823718177/dncp_block_verification.log.curr is not yet opened.
	at __randomizedtesting.SeedInfo.seed([6D29FDC01C68A451]:0)
	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RollingLogsImpl$1.append(RollingLogsImpl.java:54)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner$LogFileHandler.append(BlockPoolSliceScanner.java:819)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.updateScanStatus(BlockPoolSliceScanner.java:351)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:451)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:523)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:684)
	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:650)
	at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:60595

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:60595
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:243)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:171)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=29573, name=collection4, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=29573, name=collection4, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35954: collection already exists: awholynewstresscollection_collection4_0
	at __randomizedtesting.SeedInfo.seed([6D29FDC01C68A451]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)




Build Log:
[...truncated 11838 lines...]
   [junit4] Suite: org.apache.solr.store.hdfs.HdfsDirectoryTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/init-core-data-001
   [junit4]   2> 2159647 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true)
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 2159678 WARN  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 2159684 WARN  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2159689 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2159699 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_49066_hdfs____6z6u45/webapp
   [junit4]   2> 2159797 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 2160488 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49066
   [junit4]   2> 2160728 WARN  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2160728 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2160737 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_54235_datanode____29ma9o/webapp
   [junit4]   2> 2160815 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 2161229 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54235
   [junit4]   2> 2161330 WARN  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 2161331 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 2161340 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_39709_datanode____.pep873/webapp
   [junit4]   2> 2161491 INFO  (IPC Server handler 8 on 58514) [    ] BlockStateChange BLOCK* processReport: from storage DS-d154090c-61c0-469a-837a-d6015240a53c node DatanodeRegistration(127.0.0.1, datanodeUuid=0364a30f-d270-49b6-9e9c-ee2894bfb0f1, infoPort=54235, ipcPort=59453, storageInfo=lv=-56;cid=testClusterID;nsid=673287137;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 2161491 INFO  (IPC Server handler 8 on 58514) [    ] BlockStateChange BLOCK* processReport: from storage DS-b7ecc137-b94a-4ba3-8410-b04a90c94774 node DatanodeRegistration(127.0.0.1, datanodeUuid=0364a30f-d270-49b6-9e9c-ee2894bfb0f1, infoPort=54235, ipcPort=59453, storageInfo=lv=-56;cid=testClusterID;nsid=673287137;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 2161503 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 2161799 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39709
   [junit4]   2> 2161956 INFO  (IPC Server handler 5 on 58514) [    ] BlockStateChange BLOCK* processReport: from storage DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716 node DatanodeRegistration(127.0.0.1, datanodeUuid=461e626c-30ae-46ae-8970-8262095208ba, infoPort=39709, ipcPort=34165, storageInfo=lv=-56;cid=testClusterID;nsid=673287137;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 2161956 INFO  (IPC Server handler 5 on 58514) [    ] BlockStateChange BLOCK* processReport: from storage DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff node DatanodeRegistration(127.0.0.1, datanodeUuid=461e626c-30ae-46ae-8970-8262095208ba, infoPort=39709, ipcPort=34165, storageInfo=lv=-56;cid=testClusterID;nsid=673287137;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 2162262 INFO  (TEST-HdfsDirectoryTest.testRandomAccessWrites-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRandomAccessWrites
   [junit4]   1> Working on pass [0] contains [false]
   [junit4]   2> 2162465 INFO  (IPC Server handler 5 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW]]} size 465102
   [junit4]   2> 2162465 INFO  (IPC Server handler 1 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741825_1001 size 465102
   [junit4]   1> Working on pass [1] contains [false]
   [junit4]   2> 2167071 INFO  (IPC Server handler 0 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW]]} size 0
   [junit4]   2> 2167072 INFO  (IPC Server handler 3 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   1> Working on pass [2] contains [false]
   [junit4]   2> 2167436 INFO  (IPC Server handler 9 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741827_1003{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW]]} size 0
   [junit4]   2> 2167439 INFO  (IPC Server handler 4 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741827_1003{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   1> Working on pass [3] contains [false]
   [junit4]   2> 2177920 INFO  (IPC Server handler 1 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741828_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   2> 2177920 INFO  (IPC Server handler 1 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741828_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   1> Working on pass [4] contains [false]
   [junit4]   2> 2178940 INFO  (IPC Server handler 9 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741829_1005{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW]]} size 0
   [junit4]   2> 2178948 INFO  (IPC Server handler 7 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741829_1005{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW]]} size 0
   [junit4]   1> Working on pass [5] contains [false]
   [junit4]   2> 2183005 INFO  (IPC Server handler 6 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW]]} size 0
   [junit4]   2> 2183006 INFO  (IPC Server handler 7 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|FINALIZED]]} size 0
   [junit4]   1> Working on pass [6] contains [false]
   [junit4]   2> 2186398 INFO  (IPC Server handler 9 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741831_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW]]} size 0
   [junit4]   2> 2186398 INFO  (IPC Server handler 9 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741831_1007 size 6033306
   [junit4]   1> Working on pass [7] contains [false]
   [junit4]   2> 2193741 INFO  (IPC Server handler 8 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741832_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|FINALIZED]]} size 0
   [junit4]   2> 2193742 INFO  (IPC Server handler 0 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741832_1008 size 807769
   [junit4]   1> Working on pass [8] contains [false]
   [junit4]   2> 2197153 INFO  (IPC Server handler 5 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW]]} size 0
   [junit4]   2> 2197156 INFO  (IPC Server handler 7 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   1> Working on pass [9] contains [false]
   [junit4]   2> 2197434 INFO  (IPC Server handler 1 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW]]} size 0
   [junit4]   2> 2197434 INFO  (IPC Server handler 4 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|FINALIZED]]} size 0
   [junit4]   2> 2199346 INFO  (TEST-HdfsDirectoryTest.testRandomAccessWrites-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testRandomAccessWrites
   [junit4]   2> 2199393 INFO  (TEST-HdfsDirectoryTest.testRename-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testRename
   [junit4]   2> 2199489 INFO  (IPC Server handler 9 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741835_1011{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW]]} size 0
   [junit4]   2> 2199490 INFO  (IPC Server handler 1 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741835_1011{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|FINALIZED]]} size 0
   [junit4]   2> 2199512 INFO  (IPC Server handler 6 on 58514) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741835_1011 127.0.0.1:48818 127.0.0.1:55405 
   [junit4]   2> 2199513 INFO  (TEST-HdfsDirectoryTest.testRename-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testRename
   [junit4]   2> 2199570 INFO  (TEST-HdfsDirectoryTest.testWritingAndReadingAFile-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testWritingAndReadingAFile
   [junit4]   2> 2199688 INFO  (IPC Server handler 8 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741836_1012{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-1591d3fa-eafe-4bc3-87b2-8cb3c6acadff:NORMAL:127.0.0.1:55405|RBW], ReplicaUnderConstruction[[DISK]DS-d154090c-61c0-469a-837a-d6015240a53c:NORMAL:127.0.0.1:48818|RBW]]} size 4
   [junit4]   2> 2199688 INFO  (IPC Server handler 8 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741836_1012 size 4
   [junit4]   2> 2200109 INFO  (IPC Server handler 9 on 58514) [    ] BlockStateChange BLOCK* addToInvalidates: blk_1073741836_1012 127.0.0.1:48818 127.0.0.1:55405 
   [junit4]   2> 2200110 INFO  (TEST-HdfsDirectoryTest.testWritingAndReadingAFile-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testWritingAndReadingAFile
   [junit4]   2> 2200138 INFO  (TEST-HdfsDirectoryTest.testEOF-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEOF
   [junit4]   2> 2200460 INFO  (IPC Server handler 8 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:55405 is added to blk_1073741837_1013{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   2> 2200468 INFO  (IPC Server handler 0 on 58514) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:48818 is added to blk_1073741837_1013{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-b7ecc137-b94a-4ba3-8410-b04a90c94774:NORMAL:127.0.0.1:48818|RBW], ReplicaUnderConstruction[[DISK]DS-d06c5bcd-7c3e-4fd9-9c70-48181cc2e716:NORMAL:127.0.0.1:55405|FINALIZED]]} size 0
   [junit4]   2> 2200500 INFO  (TEST-HdfsDirectoryTest.testEOF-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testEOF
   [junit4]   2> 2200501 WARN  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 2200518 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 2200524 WARN  (554066667@qtp-1316213847-0 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39709) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
   [junit4]   2> 2200524 WARN  (554066667@qtp-1316213847-0 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:39709) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
   [junit4]   2> 2200538 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:58514) [    ] o.a.h.h.s.d.DataNode BPOfferService for Block pool BP-343369792-10.41.0.5-1460823718177 (Datanode Uuid 461e626c-30ae-46ae-8970-8262095208ba) service to localhost/127.0.0.1:58514 interrupted
   [junit4]   2> 2200538 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data3/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data4/]]  heartbeating to localhost/127.0.0.1:58514) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-343369792-10.41.0.5-1460823718177 (Datanode Uuid 461e626c-30ae-46ae-8970-8262095208ba) service to localhost/127.0.0.1:58514
   [junit4]   2> 2200544 WARN  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.h.s.d.DirectoryScanner DirectoryScanner: shutdown has been called
   [junit4]   2> 2200550 WARN  (DataNode: [[[DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data1/, [DISK]file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data2/]]  heartbeating to localhost/127.0.0.1:58514) [    ] o.a.h.h.s.d.DataNode Ending block pool service for: Block pool BP-343369792-10.41.0.5-1460823718177 (Datanode Uuid 0364a30f-d270-49b6-9e9c-ee2894bfb0f1) service to localhost/127.0.0.1:58514
   [junit4]   2> 2200552 WARN  (Thread-8241) [    ] o.a.h.h.s.d.BlockPoolSliceScanner RuntimeException during BlockPoolScanner.scan()
   [junit4]   2> java.lang.IllegalStateException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data1/current/BP-343369792-10.41.0.5-1460823718177/dncp_block_verification.log.curr is not yet opened.
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RollingLogsImpl$1.append(RollingLogsImpl.java:54)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner$LogFileHandler.append(BlockPoolSliceScanner.java:819)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.updateScanStatus(BlockPoolSliceScanner.java:351)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:451)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:523)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:684)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:650)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Apr 16, 2016 12:22:39 PM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Thread-8241,5,TGRP-HdfsDirectoryTest]
   [junit4]   2> java.lang.IllegalStateException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data1/current/BP-343369792-10.41.0.5-1460823718177/dncp_block_verification.log.curr is not yet opened.
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([6D29FDC01C68A451]:0)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RollingLogsImpl$1.append(RollingLogsImpl.java:54)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner$LogFileHandler.append(BlockPoolSliceScanner.java:819)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.updateScanStatus(BlockPoolSliceScanner.java:351)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:451)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:523)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:684)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:650)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 
   [junit4]   2> 2200585 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 2200597 WARN  (981596650@qtp-1665797956-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54235) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
   [junit4]   2> 2200597 WARN  (981596650@qtp-1665797956-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54235) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
   [junit4]   2> 2200706 WARN  (org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@2e98b70e) [    ] o.a.h.h.s.b.DecommissionManager Monitor interrupted: java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 2200722 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.m.log Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0
   [junit4]   2> 2200722 WARN  (1498840169@qtp-1687881667-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49066) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false. Rechecking.
   [junit4]   2> 2200723 WARN  (1498840169@qtp-1687881667-1 - Acceptor0 HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:49066) [    ] o.a.h.h.HttpServer2 HttpServer Acceptor: isRunning is false
   [junit4]   2> 2200894 ERROR (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.h.m.l.MethodMetric Error invoking method getBlocksTotal
   [junit4]   2> java.lang.reflect.InvocationTargetException
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MethodMetric$2.snapshot(MethodMetric.java:111)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MethodMetric.snapshot(MethodMetric.java:144)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MetricsRegistry.snapshot(MetricsRegistry.java:387)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.MetricsSourceBuilder$1.getMetrics(MetricsSourceBuilder.java:79)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMetrics(MetricsSourceAdapter.java:195)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.updateJmxCache(MetricsSourceAdapter.java:172)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.getMBeanInfo(MetricsSourceAdapter.java:151)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassName(DefaultMBeanServerInterceptor.java:1804)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.safeGetClassName(DefaultMBeanServerInterceptor.java:1595)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.checkMBeanPermission(DefaultMBeanServerInterceptor.java:1813)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.exclusiveUnregisterMBean(DefaultMBeanServerInterceptor.java:430)
   [junit4]   2> 	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.unregisterMBean(DefaultMBeanServerInterceptor.java:415)
   [junit4]   2> 	at com.sun.jmx.mbeanserver.JmxMBeanServer.unregisterMBean(JmxMBeanServer.java:546)
   [junit4]   2> 	at org.apache.hadoop.metrics2.util.MBeans.unregister(MBeans.java:81)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stopMBeans(MetricsSourceAdapter.java:227)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSourceAdapter.stop(MetricsSourceAdapter.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stopSources(MetricsSystemImpl.java:461)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.stop(MetricsSystemImpl.java:212)
   [junit4]   2> 	at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.shutdown(MetricsSystemImpl.java:592)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdownInstance(DefaultMetricsSystem.java:72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2> 	at org.apache.solr.store.hdfs.HdfsDirectoryTest.afterClass(HdfsDirectoryTest.java:66)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 2200896 INFO  (SUITE-HdfsDirectoryTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Apr 16, 2016 4:22:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=RandomSimilarity(queryNorm=true,coord=no): {}, locale=ar-TN, timezone=America/Grand_Turk
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=236149672,total=492830720
   [junit4]   2> NOTE: All tests run in this JVM: [FileUtilsTest, BigEndianAscendingWordSerializerTest, SchemaVersionSpecificBehaviorTest, BlockCacheTest, TestConfigReload, TestSerializedLuceneMatchVersion, TestSystemIdResolver, TestWordDelimiterFilterFactory, ZkSolrClientTest, OpenExchangeRatesOrgProviderTest, ShardSplitTest, DistributedIntervalFacetingTest, TestConfigSetImmutable, IgnoreCommitOptimizeUpdateProcessorFactoryTest, CollectionReloadTest, ClusterStateUpdateTest, UniqFieldsUpdateProcessorFactoryTest, SimplePostToolTest, SpellPossibilityIteratorTest, TestRestoreCore, CoreMergeIndexesAdminHandlerTest, ZkControllerTest, DirectSolrSpellCheckerTest, ZkCLITest, AnalysisErrorHandlingTest, TestMiniSolrCloudClusterBase, MigrateRouteKeyTest, CSVRequestHandlerTest, TestClassicSimilarityFactory, NotRequiredUniqueKeyTest, HdfsChaosMonkeySafeLeaderTest, ChaosMonkeySafeLeaderTest, TestOrdValues, TestSchemaManager, TestQuerySenderListener, CdcrReplicationHandlerTest, TestSmileRequest, DistributedFacetPivotSmallTest, TestDistributedGrouping, WordBreakSolrSpellCheckerTest, FastVectorHighlighterTest, TestSolrConfigHandlerConcurrent, TestExactStatsCache, TestFiltering, TestAuthenticationFramework, TermsComponentTest, TestQuerySenderNoQuery, DistanceUnitsTest, OverseerStatusTest, SparseHLLTest, TestPartialUpdateDeduplication, MultiTermTest, TestStressVersions, HdfsDirectoryFactoryTest, TestComplexPhraseQParserPlugin, RemoteQueryErrorTest, TestLMJelinekMercerSimilarityFactory, TlogReplayBufferedWhileIndexingTest, TestPerFieldSimilarityClassic, TestEmbeddedSolrServerConstructors, TestInitParams, TestNamedUpdateProcessors, HdfsDirectoryTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=HdfsDirectoryTest -Dtests.seed=6D29FDC01C68A451 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ar-TN -Dtests.timezone=America/Grand_Turk -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | HdfsDirectoryTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=14245, name=Thread-8241, state=RUNNABLE, group=TGRP-HdfsDirectoryTest]
   [junit4]    > Caused by: java.lang.IllegalStateException: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.store.hdfs.HdfsDirectoryTest_6D29FDC01C68A451-001/tempDir-001/hdfsBaseDir/data/data1/current/BP-343369792-10.41.0.5-1460823718177/dncp_block_verification.log.curr is not yet opened.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([6D29FDC01C68A451]:0)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.RollingLogsImpl$1.append(RollingLogsImpl.java:54)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner$LogFileHandler.append(BlockPoolSliceScanner.java:819)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.updateScanStatus(BlockPoolSliceScanner.java:351)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyBlock(BlockPoolSliceScanner.java:451)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.verifyFirstBlock(BlockPoolSliceScanner.java:523)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scan(BlockPoolSliceScanner.java:684)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner.scanBlockPoolSlice(BlockPoolSliceScanner.java:650)
   [junit4]    > 	at org.apache.hadoop.hdfs.server.datanode.DataBlockScanner.run(DataBlockScanner.java:101)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [211/587 (1!)] on J1 in 51.12s, 4 tests, 1 error <<< FAILURES!

[...truncated 211 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/init-core-data-001
   [junit4]   2> 2080587 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2080587 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2080590 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2080590 INFO  (Thread-6931) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2080590 INFO  (Thread-6931) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2080693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.ZkTestServer start zk server on port:60836
   [junit4]   2> 2080693 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2080702 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2080725 INFO  (zkCallback-1541-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6875b061 name:ZooKeeperConnection Watcher:127.0.0.1:60836 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2080725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2080725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2080725 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2080735 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2080745 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2080757 INFO  (zkCallback-1542-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7e060b0b name:ZooKeeperConnection Watcher:127.0.0.1:60836/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2080761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2080761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2080761 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 2080768 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 2080769 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 2080770 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 2080771 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2080771 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2080773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2080773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 2080775 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2080776 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2080777 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2080778 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2080780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2080780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2080782 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2080782 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 2080785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2080785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2080786 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2080786 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2080788 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2080788 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2080790 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2080790 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2080792 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2080792 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2080802 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2080804 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2080809 INFO  (zkCallback-1543-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@697c045e name:ZooKeeperConnection Watcher:127.0.0.1:60836 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2080809 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2080809 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2080809 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2080812 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2080813 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2080820 INFO  (zkCallback-1544-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6e5af73 name:ZooKeeperConnection Watcher:127.0.0.1:60836/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2080825 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2080825 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2080825 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 2080825 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 2080838 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 2080839 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 2080839 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 2080840 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 2080840 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2080841 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2080841 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 2080842 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 2080849 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 2080849 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 2080850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 2080850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 2080851 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 2080851 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 2080852 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2080852 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2080853 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 2080853 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 2080861 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 2080861 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 2080863 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 2080863 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 2081136 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/cores/collection1
   [junit4]   2> 2081137 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 2081139 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@214f88a0{/,null,AVAILABLE}
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.e.j.s.ServerConnector Started ServerConnector@7dab91b6{HTTP/1.1,[http/1.1]}{127.0.0.1:39528}
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.e.j.s.Server Started @2088126ms
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/tempDir-001/control/data, hostContext=/, hostPort=39528, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/cores}
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001'
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2081140 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2081143 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2081153 INFO  (zkCallback-1545-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4d2726ea name:ZooKeeperConnection Watcher:127.0.0.1:60836/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2081153 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2081153 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2081155 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2081155 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/solr.xml
   [junit4]   2> 2081159 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/cores
   [junit4]   2> 2081159 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.CoreContainer New CoreContainer 461437162
   [junit4]   2> 2081159 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001]
   [junit4]   2> 2081159 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/lib
   [junit4]   2> 2081159 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2081160 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2081161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2081161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2081161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:60836/solr
   [junit4]   2> 2081161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2081161 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2081162 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2081162 INFO  (zkCallback-1548-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@225ebb8 name:ZooKeeperConnection Watcher:127.0.0.1:60836 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2081163 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2081163 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2081164 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$16/933838597@9c59634
   [junit4]   2> 2081164 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2081166 INFO  (zkCallback-1549-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5c17b295 name:ZooKeeperConnection Watcher:127.0.0.1:60836/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2081166 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2081167 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2081168 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 2081169 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 2081170 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 2081171 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 2081171 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 2081172 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 2081173 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 2081173 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 2081174 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2081174 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 2081180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2081181 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 2081185 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2081185 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95736537992200198-127.0.0.1:39528_-n_0000000000
   [junit4]   2> 2081186 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39528_
   [junit4]   2> 2081186 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 2081186 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.Overseer Overseer (id=95736537992200198-127.0.0.1:39528_-n_0000000000) starting
   [junit4]   2> 2081188 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2081196 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2081197 INFO  (OverseerStateUpdate-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2081197 INFO  (OverseerCollectionConfigSetProcessor-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 2081199 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39528_
   [junit4]   2> 2081200 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:39528_
   [junit4]   2> 2081202 INFO  (OverseerStateUpdate-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 2081202 INFO  (zkCallback-1549-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2081205 INFO  (zkCallback-1549-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2081210 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2081210 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2081210 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/cores
   [junit4]   2> 2081211 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2081211 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/cores/collection1
   [junit4]   2> 2081211 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 2081253 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 2081254 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[6D29FDC01C68A451]) [n:127.0.0.1:39528_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 2081255 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2081255 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2081297 INFO  (OverseerStateUpdate-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:39528",
   [junit4]   2>   "node_name":"127.0.0.1:39528_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2081297 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 2081304 INFO  (OverseerStateUpdate-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:39528",
   [junit4]   2>   "node_name":"127.0.0.1:39528_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2081308 INFO  (OverseerStateUpdate-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 2081313 INFO  (OverseerStateUpdate-95736537992200198-127.0.0.1:39528_-n_0000000000) [n:127.0.0.1:39528_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2081333 INFO  (zkCallback-1549-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2082304 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2082304 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 2082304 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 2082304 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 2082305 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 2082305 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.CollectionsAPIDistributedZkTest_6D29FDC01C68A451-001/control-001/cores/collection1'
   [junit4]   2> 2082305 INFO  (coreLoadExecutor-4347-thread-1-processing-n:127.0.0.1:39528_) [n:127.0.0.1:39528_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx

[...truncated too long message...]

junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 5920134 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[6D29FDC01C68A451]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> Apr 16, 2016 5:24:38 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=PostingsFormat(name=Memory doPackFST= false), range_facet_l_dv=BlockTreeOrds(blocksize=128), _version_=PostingsFormat(name=Memory doPackFST= false), multiDefault=PostingsFormat(name=Memory doPackFST= true), a_t=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Memory doPackFST= false), a_i1=PostingsFormat(name=Memory doPackFST= false), id=BlockTreeOrds(blocksize=128), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= false), text=PostingsFormat(name=LuceneVarGapDocFreqInterval), range_facet_l=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Memory), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=539, maxMBSortInHeap=4.863911445567977, sim=ClassicSimilarity, locale=lv-LV, timezone=America/Menominee
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=115093408,total=496500736
   [junit4]   2> NOTE: All tests run in this JVM: [TestManagedResource, TestZkChroot, TestSolrQueryParserDefaultOperatorResource, TestUseDocValuesAsStored, TestSuggestSpellingConverter, TestStressUserVersions, DistributedSuggestComponentTest, ParsingFieldUpdateProcessorsTest, TestMissingGroups, TestFieldCollectionResource, TestPerFieldSimilarity, TestReversedWildcardFilterFactory, RuleEngineTest, TestIndexSearcher, TestSizeLimitedDistributedMap, DistributedFacetPivotLongTailTest, EnumFieldTest, TestCloudManagedSchema, SolrCloudExampleTest, SearchHandlerTest, SliceStateTest, DirectoryFactoryTest, TestStressRecovery, TestRawTransformer, TestElisionMultitermQuery, HdfsRecoveryZkTest, TestSearcherReuse, HdfsWriteToMultipleCollectionsTest, TestScoreJoinQPNoScore, TestSQLHandler, TestLRUStatsCache, TestJmxIntegration, AddSchemaFieldsUpdateProcessorFactoryTest, TestJmxMonitoredMap, RequestLoggingTest, TestSha256AuthenticationProvider, TestConfigSetsAPIZkFailure, BasicDistributedZk2Test, CheckHdfsIndexTest, QueryParsingTest, TestReRankQParserPlugin, ClusterStateTest, TestCollationField, TestExpandComponent, TestSearchPerf, TestDefaultSearchFieldResource, DeleteLastCustomShardedReplicaTest, TestHashQParserPlugin, TestIndexingPerformance, RankQueryTest, StatsComponentTest, TriLevelCompositeIdRoutingTest, TestTolerantUpdateProcessorCloud, HighlighterConfigTest, CleanupOldIndexTest, SuggesterTSTTest, TestPostingsSolrHighlighter, DirectSolrConnectionTest, ShowFileRequestHandlerTest, TestDFISimilarityFactory, TestImplicitCoreProperties, BaseCdcrDistributedZkTest, SuggesterTest, TestSolrIndexConfig, NoCacheHeaderTest, JavabinLoaderTest, LeaderInitiatedRecoveryOnCommitTest, TestJsonFacets, TestDistributedMissingSort, TestTestInjection, FieldMutatingUpdateProcessorTest, AnalyticsQueryTest, TestPerFieldSimilarityWithDefaultOverride, TestCharFilters, TermVectorComponentDistributedTest, ForceLeaderTest, CurrencyFieldXmlFileTest, DistribJoinFromCollectionTest, CdcrRequestHandlerTest, TestSolr4Spatial2, TestSweetSpotSimilarityFactory, TestWriterPerf, SharedFSAutoReplicaFailoverTest, CloudMLTQParserTest, CoreAdminCreateDiscoverTest, TestUtils, ZkStateReaderTest, BadIndexSchemaTest, RegexBoostProcessorTest, TestXIncludeConfig, RecoveryZkTest, TestSchemaSimilarityResource, QueryResultKeyTest, TestTolerantSearch, BasicDistributedZkTest, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, SyncSliceTest, OpenCloseCoreStressTest, OverseerTest, LeaderElectionIntegrationTest, ShardRoutingTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, LeaderElectionTest, TestRandomDVFaceting, AutoCommitTest, TestRecovery, TestRealTimeGet, TestStressReorder, TestJoin, TestReload, TestRangeQuery, SimpleFacetsTest, TestGroupingSearch, BasicFunctionalityTest, DirectUpdateHandlerTest, HighlighterTest, CurrencyFieldOpenExchangeTest, TestCoreDiscovery, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SuggesterFSTTest, SolrRequestParserTest, DocValuesTest, XsltUpdateRequestHandlerTest, TestSurroundQueryParser, LukeRequestHandlerTest, RequestHandlersTest, TestValueSourceCache, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestArbitraryIndexDir, UpdateRequestProcessorFactoryTest, TestCSVResponseWriter, TestAnalyzedSuggestions, BinaryUpdateRequestHandlerTest, TestBinaryResponseWriter, AlternateDirectoryTest, SolrIndexConfigTest, SampleTest, TestDocSet, NumericFieldsTest, TestConfig, OutputWriterTest, ExternalFileFieldSortTest, TestPhraseSuggestions, SynonymTokenizerTest, EchoParamsTest, TestDFRSimilarityFactory, TestIBSimilarityFactory, TestFastWriter, TestFastLRUCache, ChaosMonkeyNothingIsSafeTest, AliasIntegrationTest, CollectionTooManyReplicasTest, ConcurrentDeleteAndCreateCollectionTest, ConnectionManagerTest, CustomCollectionTest, DistribDocExpirationUpdateProcessorTest, DistributedQueueTest, DistributedVersionInfoTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, SSLMigrationTest, SharedFSAutoReplicaFailoverUtilsTest, SimpleCollectionCreateDeleteTest, TestCloudDeleteByQuery, TestCloudPivotFacet, TestCollectionAPI, TestDistribDocBasedVersion, TestDownShardTolerantSearch, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestRandomRequestDistribution, TestShortCircuitedRequests, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [479/587 (4!)] on J2 in 464.15s, 1 test, 1 error <<< FAILURES!

[...truncated 391 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 587 suites (10 ignored), 2366 tests, 1 suite-level error, 2 errors, 1 failure, 31 ignored (16 assumptions) [seed: 6D29FDC01C68A451]

Total time: 313 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 39 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/39/

4 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=8945, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=8945, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:50419: collection already exists: awholynewstresscollection_collection4_0
	at __randomizedtesting.SeedInfo.seed([A1D15E71442F8FE5]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=66270, name=collection0, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=66270, name=collection0, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:34611/wt_qrn/xk: collection already exists: awholynewstresscollection_collection0_0
	at __randomizedtesting.SeedInfo.seed([A1D15E71442F8FE5]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1595)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1616)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([A1D15E71442F8FE5:CA9EFE0C3D2052DF]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
[index.20160415211150995, replication.properties, index.properties, index.20160415211155931] expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: [index.20160415211150995, replication.properties, index.properties, index.20160415211155931] expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([A1D15E71442F8FE5:7A7A5EB74107E656]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.handler.TestReplicationHandler.checkForSingleIndex(TestReplicationHandler.java:824)
	at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:791)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11879 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/init-core-data-001
   [junit4]   2> 1905550 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[A1D15E71442F8FE5]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 1905550 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[A1D15E71442F8FE5]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1905579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1905586 INFO  (Thread-4313) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1905586 INFO  (Thread-4313) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1905682 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkTestServer start zk server on port:51315
   [junit4]   2> 1905683 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1905683 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1905686 INFO  (zkCallback-956-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@21e25c6a name:ZooKeeperConnection Watcher:127.0.0.1:51315 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1905686 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1905686 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1905686 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1905688 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1905689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1905689 INFO  (zkCallback-957-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@75d8e8b5 name:ZooKeeperConnection Watcher:127.0.0.1:51315/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1905689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1905689 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1905690 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1905840 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1905841 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1905842 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1905844 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1905844 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1905846 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1905847 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1905859 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1905859 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1905860 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1905860 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1905861 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1905861 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1905861 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1905861 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1905862 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1905862 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1905863 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1905863 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1905863 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1905864 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1905864 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1905864 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1905865 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1905865 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1905928 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1
   [junit4]   2> 1905930 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@74f33693{/,null,AVAILABLE}
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4a7b4a03{HTTP/1.1,[http/1.1]}{127.0.0.1:40741}
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.Server Started @1913071ms
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/tempDir-001/control/data, hostContext=/, hostPort=40741, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores}
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001'
   [junit4]   2> 1905931 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1905932 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1905932 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1905939 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1905978 INFO  (zkCallback-958-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4729d083 name:ZooKeeperConnection Watcher:127.0.0.1:51315/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1905982 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1905983 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1905987 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1905987 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/solr.xml
   [junit4]   2> 1905991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores
   [junit4]   2> 1905991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CoreContainer New CoreContainer 200271399
   [junit4]   2> 1905991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001]
   [junit4]   2> 1905991 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/lib
   [junit4]   2> 1905991 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1905992 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1905993 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1905993 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1905993 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51315/solr
   [junit4]   2> 1905993 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1905993 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1905993 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1905994 INFO  (zkCallback-961-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2d9715eb name:ZooKeeperConnection Watcher:127.0.0.1:51315 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1905995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1905995 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1905996 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$29/526577254@2904576d
   [junit4]   2> 1905997 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1905998 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@64ed056b name:ZooKeeperConnection Watcher:127.0.0.1:51315/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1905998 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1905999 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1906003 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1906004 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1906005 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1906005 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1906006 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1906007 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1906007 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1906007 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1906008 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1906008 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1906009 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1906009 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1906010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1906010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95731938238922756-127.0.0.1:40741_-n_0000000000
   [junit4]   2> 1906010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:40741_
   [junit4]   2> 1906010 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1906011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer Overseer (id=95731938238922756-127.0.0.1:40741_-n_0000000000) starting
   [junit4]   2> 1906011 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1906013 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1906014 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1906014 INFO  (OverseerCollectionConfigSetProcessor-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1906015 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40741_
   [junit4]   2> 1906016 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:40741_
   [junit4]   2> 1906027 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1906030 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1906034 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 1906039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1906039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1906043 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores
   [junit4]   2> 1906043 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1906043 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1
   [junit4]   2> 1906043 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1906070 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1906070 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1906070 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1906070 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:40741_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1906094 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40741",
   [junit4]   2>   "node_name":"127.0.0.1:40741_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1906100 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40741",
   [junit4]   2>   "node_name":"127.0.0.1:40741_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1906100 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1906106 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1906107 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1906108 INFO  (zkCallback-962-thread-2-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1907110 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1907110 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1907111 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1907111 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1907111 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1907111 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1'
   [junit4]   2> 1907112 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1907112 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1907113 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1907116 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1907127 WARN  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1907128 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1907143 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1907146 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1907298 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1907303 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1907303 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1907309 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1907310 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1907312 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1907312 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1907313 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1907313 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1907313 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1907313 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data/]
   [junit4]   2> 1907313 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@730d155b
   [junit4]   2> 1907314 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data
   [junit4]   2> 1907314 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1907314 WARN  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1907314 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data/index
   [junit4]   2> 1907315 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=49, maxMergeAtOnceExplicit=38, maxMergedSegmentMB=80.998046875, floorSegmentMB=0.25390625, forceMergeDeletesPctAllowed=8.208700979100245, segmentsPerTier=5.0, maxCFSSegmentSizeMB=1.5927734375, noCFSRatio=0.8012100385426278
   [junit4]   2> 1907315 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@387a91a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@160c0818),segFN=segments_1,generation=1}
   [junit4]   2> 1907315 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1907315 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data/
   [junit4]   2> 1907315 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1907318 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1907318 WARN  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1907319 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1907320 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1907320 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1907321 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1907323 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1907323 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1907323 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1907323 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1907324 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1907324 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1907324 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=9, maxMergeAtOnceExplicit=3, maxMergedSegmentMB=0.5546875, floorSegmentMB=0.7890625, forceMergeDeletesPctAllowed=0.9103868411678484, segmentsPerTier=50.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.217491069190051
   [junit4]   2> 1907325 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@387a91a7 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@160c0818),segFN=segments_1,generation=1}
   [junit4]   2> 1907325 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1907325 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@57be590c[collection1] main]
   [junit4]   2> 1907325 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1907325 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1907326 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1907327 INFO  (searcherExecutor-2928-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@57be590c[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1907327 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1907327 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1907327 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1907327 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1531711013639946240
   [junit4]   2> 1907329 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1531711013639946240
   [junit4]   2> 1907329 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1907330 INFO  (coreLoadExecutor-2927-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1907338 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:40741 collection:control_collection shard:shard1
   [junit4]   2> 1907339 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1907341 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95731938238922756-core_node1-n_0000000000
   [junit4]   2> 1907347 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1907348 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1907348 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1907348 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:40741/collection1/
   [junit4]   2> 1907348 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1907348 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:40741/collection1/ has no replicas
   [junit4]   2> 1907349 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1907350 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95731938238922756-core_node1-n_0000000000
   [junit4]   2> 1907351 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:40741/collection1/ shard1
   [junit4]   2> 1907358 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1907363 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:40741",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 1907464 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1907502 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:40741/collection1/ and leader is http://127.0.0.1:40741/collection1/
   [junit4]   2> 1907502 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40741
   [junit4]   2> 1907502 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1907502 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 1907502 INFO  (coreZkRegister-2921-thread-1-processing-n:127.0.0.1:40741_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:40741_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1907506 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40741",
   [junit4]   2>   "node_name":"127.0.0.1:40741_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 1907506 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:40741",
   [junit4]   2>   "node_name":"127.0.0.1:40741_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1907571 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.s.i.CloudSolrClient Final constructed zkHost string: 127.0.0.1:51315/solr
   [junit4]   2> 1907572 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1907579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1907579 INFO  (zkCallback-964-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2a39f7cf name:ZooKeeperConnection Watcher:127.0.0.1:51315/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1907579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1907579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1907579 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1907580 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1907580 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1907611 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1907614 INFO  (zkCallback-964-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1907766 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/cores/collection1
   [junit4]   2> 1907769 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001
   [junit4]   2> 1907769 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1907773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@70337fc1{/,null,AVAILABLE}
   [junit4]   2> 1907773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.ServerConnector Started ServerConnector@70790ebe{HTTP/1.1,[http/1.1]}{127.0.0.1:50419}
   [junit4]   2> 1907773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.e.j.s.Server Started @1914913ms
   [junit4]   2> 1907773 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=50419, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/cores}
   [junit4]   2> 1907774 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1907774 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001'
   [junit4]   2> 1907774 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1907774 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1907774 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1907777 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1907778 INFO  (zkCallback-965-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@fc198d9 name:ZooKeeperConnection Watcher:127.0.0.1:51315/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1907778 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1907779 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1907780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1907780 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/solr.xml
   [junit4]   2> 1907783 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/cores
   [junit4]   2> 1907783 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CoreContainer New CoreContainer 1111178328
   [junit4]   2> 1907783 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001]
   [junit4]   2> 1907784 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/lib
   [junit4]   2> 1907784 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1907784 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1907785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1907785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1907785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51315/solr
   [junit4]   2> 1907785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1907785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1907785 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1907786 INFO  (zkCallback-968-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@55aa5162 name:ZooKeeperConnection Watcher:127.0.0.1:51315 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1907787 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1907787 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1907788 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.ZkController Added new OnReconnect listener org.apache.solr.cloud.ZkController$$Lambda$29/526577254@ec58a91
   [junit4]   2> 1907788 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1907789 INFO  (zkCallback-969-thread-1-processing-n:127.0.0.1:50419_) [n:127.0.0.1:50419_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@a54d021 name:ZooKeeperConnection Watcher:127.0.0.1:51315/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1907789 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1907793 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1907794 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1907795 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1907798 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95731938238922760-127.0.0.1:50419_-n_0000000001
   [junit4]   2> 1907799 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95731938238922756-127.0.0.1:40741_-n_0000000000 to know if I could be the leader
   [junit4]   2> 1907799 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.ZkController Publish node=127.0.0.1:50419_ as DOWN
   [junit4]   2> 1907810 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:50419_"} current state version: 3
   [junit4]   2> 1907810 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40741_-n_0000000000) [n:127.0.0.1:40741_    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:50419_
   [junit4]   2> 1907935 INFO  (zkCallback-964-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1907938 INFO  (zkCallback-969-thread-1-processing-n:127.0.0.1:50419_) [n:127.0.0.1:50419_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1907938 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1908801 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50419_
   [junit4]   2> 1908801 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:50419_
   [junit4]   2> 1908802 INFO  (zkCallback-964-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1908802 INFO  (zkCallback-969-thread-1-processing-n:127.0.0.1:50419_) [n:127.0.0.1:50419_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1908802 INFO  (zkCallback-969-thread-1-processing-n:127.0.0.1:50419_) [n:127.0.0.1:50419_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1908802 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1908810 INFO  (zkCallback-962-thread-1-processing-n:127.0.0.1:40741_) [n:127.0.0.1:40741_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1908815 INFO  (zkCallback-964-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1908886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1908886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1908886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/cores
   [junit4]   2> 1908909 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1908909 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_A1D15E71442F8FE5-001/shard-1-001/cores/collection1
   [junit4]   2> 1908909 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[A1D15E71442F8FE5]) [n:127.0.0.1:50419_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1908919 INFO  (coreLoadExecutor-2938-thread-1-processing-n:127.0.0.1:50419_) [n:127.0.0.1:50419_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1908919 INFO  (coreLoadExecutor-2938-thread-1-processing-n:127.0.0.1:50419_) [n:127.0.0.1:50419_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1908920 INFO  (OverseerStateUpdate-95731938238922756-127.0.0.1:40

[...truncated too long message...]

it4]   2> 6111354 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 6111354 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 6111354 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 6111354 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 6111356 INFO  (zkCallback-4513-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 6111356 INFO  (zkCallback-4513-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 6111406 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 6111406 INFO  (zkCallback-4513-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 6111407 INFO  (zkCallback-4513-thread-2) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 6111409 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45676 45676
   [junit4]   2> 6111697 INFO  (Thread-50310) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:45676 45676
   [junit4]   2> 6111707 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[A1D15E71442F8FE5]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelay
   [junit4]   2> 6111707 INFO  (SUITE-ZkStateReaderTest-seed#[A1D15E71442F8FE5]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.overseer.ZkStateReaderTest_A1D15E71442F8FE5-001
   [junit4]   2> Apr 15, 2016 10:00:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=932, maxMBSortInHeap=6.658889566033512, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=uk-UA, timezone=US/Indiana-Starke
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=162115440,total=498597888
   [junit4]   2> NOTE: All tests run in this JVM: [TestSort, TestJsonFacets, HardAutoCommitTest, UnloadDistributedZkTest, TestCollationFieldDocValues, SearchHandlerTest, TestCharFilters, TestCollectionAPI, TestSuggestSpellingConverter, TestStressReorder, BlockJoinFacetRandomTest, RegexBoostProcessorTest, TestElisionMultitermQuery, PluginInfoTest, JavabinLoaderTest, TestSolr4Spatial2, TestPHPSerializedResponseWriter, TestExpandComponent, TestManagedResource, RequestLoggingTest, DistributedSuggestComponentTest, HighlighterConfigTest, SolrCoreCheckLockOnStartupTest, SolrCloudExampleTest, ParsingFieldUpdateProcessorsTest, TestExceedMaxTermLength, CdcrRequestHandlerTest, MultiThreadedOCPTest, SuggesterWFSTTest, TestSearcherReuse, TestSweetSpotSimilarityFactory, ShowFileRequestHandlerTest, TestIndexingPerformance, TestDefaultSearchFieldResource, BaseCdcrDistributedZkTest, AddSchemaFieldsUpdateProcessorFactoryTest, BadIndexSchemaTest, BitVectorTest, TestSha256AuthenticationProvider, DistribJoinFromCollectionTest, TestCloudManagedSchema, RequestHandlersTest, LukeRequestHandlerTest, TolerantUpdateProcessorTest, OverseerCollectionConfigSetProcessorTest, DeleteLastCustomShardedReplicaTest, TestSQLHandler, HdfsBasicDistributedZkTest, OverseerTaskQueueTest, TestDownShardTolerantSearch, TestRawTransformer, TestRecovery, BlockDirectoryTest, TestSolrQueryParserDefaultOperatorResource, AutoCommitTest, TermVectorComponentDistributedTest, TestIndexSearcher, TestHashQParserPlugin, TestReplicationHandlerBackup, TestBulkSchemaAPI, CollectionStateFormat2Test, RecoveryZkTest, TestReversedWildcardFilterFactory, TestReRankQParserPlugin, XmlUpdateRequestHandlerTest, TestLeaderInitiatedRecoveryThread, BasicDistributedZk2Test, CollectionsAPISolrJTest, TestManagedStopFilterFactory, TriLevelCompositeIdRoutingTest, DistributedFacetPivotWhiteBoxTest, CoreAdminCreateDiscoverTest, TestPerFieldSimilarityWithDefaultOverride, TestRequestForwarding, TestJmxIntegration, NoCacheHeaderTest, HdfsThreadLeakTest, CurrencyFieldXmlFileTest, TestCSVResponseWriter, TestSizeLimitedDistributedMap, PrimitiveFieldTypeTest, TestXIncludeConfig, TestCloudDeleteByQuery, TestSearchPerf, CircularListTest, CloudMLTQParserTest, AnalyticsQueryTest, TestExtendedDismaxParser, SuggestComponentContextFilterQueryTest, TestScoreJoinQPNoScore, TestFieldTypeResource, TestNRTOpen, DirectSolrConnectionTest, FieldMutatingUpdateProcessorTest, DirectoryFactoryTest, SolrTestCaseJ4Test, BasicDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, FullSolrCloudDistribCmdsTest, TestRandomFaceting, LeaderElectionTest, TestZkChroot, TestRandomDVFaceting, TestFaceting, TestHashPartitioner, TestRealTimeGet, TestJoin, TestReload, TestStressVersions, TestRangeQuery, SimpleFacetsTest, TestSolr4Spatial, StatsComponentTest, TestGroupingSearch, SolrCmdDistributorTest, PeerSyncTest, ConvertedLegacyTest, TestLazyCores, DirectUpdateHandlerTest, HighlighterTest, SoftAutoCommitTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, CoreAdminHandlerTest, SolrRequestParserTest, SuggesterTSTTest, SuggesterTest, TestStressLucene, TestTrie, SpatialFilterTest, TestCSVLoader, PolyFieldTest, WordBreakSolrSpellCheckerTest, SchemaVersionSpecificBehaviorTest, TestUpdate, TestAtomicUpdateErrorCases, QueryEqualityTest, DirectUpdateHandlerOptimizeTest, DefaultValueUpdateProcessorTest, StatelessScriptUpdateProcessorFactoryTest, TestRemoteStreaming, DocValuesMultiTest, TestSolrDeletionPolicy1, SolrInfoMBeanTest, DebugComponentTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, TestWriterPerf, DirectSolrSpellCheckerTest, TestOmitPositions, FileBasedSpellCheckerTest, DocumentBuilderTest, TestValueSourceCache, TermVectorComponentTest, MoreLikeThisHandlerTest, FieldAnalysisRequestHandlerTest, TestSolrQueryParser, TestArbitraryIndexDir, IndexSchemaTest, TestCollationField, ReturnFieldsTest, UpdateRequestProcessorFactoryTest, JSONWriterTest, QueryParsingTest, TestAnalyzedSuggestions, JsonLoaderTest, BinaryUpdateRequestHandlerTest, TestPartialUpdateDeduplication, TestBinaryResponseWriter, TestQuerySenderListener, UpdateParamsTest, TestSolrIndexConfig, TestQuerySenderNoQuery, CopyFieldTest, BadComponentTest, TestStressRecovery, TestMergePolicyConfig, TestSolrDeletionPolicy2, MultiTermTest, SampleTest, TestBinaryField, OutputWriterTest, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestPostingsSolrHighlighter, TestLuceneMatchVersion, TestPhraseSuggestions, TestCodecSupport, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestBM25SimilarityFactory, TestFastWriter, TestJmxMonitoredMap, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, TestFastLRUCache, DateMathParserTest, ChaosMonkeyNothingIsSafeTest, TestSystemIdResolver, DateFieldTest, DOMUtilTest, ClusterStateTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, TestRTGBase, CursorPagingTest, TestDistributedMissingSort, TestHighlightDedupGrouping, TestSimpleTrackingShardHandler, TestTolerantSearch, TestEmbeddedSolrServerConstructors, TestJettySolrRunner, AssignTest, AsyncCallRequestStatusResponseTest, CdcrReplicationDistributedZkTest, SSLMigrationTest, SharedFSAutoReplicaFailoverTest, TestExclusionRuleCollectionAccess, TestRandomRequestDistribution, TestTolerantUpdateProcessorCloud, HdfsBasicDistributedZk2Test, HdfsCollectionsAPIDistributedZkTest, ZkStateReaderTest]
   [junit4] Completed [476/587 (4!)] on J2 in 5.18s, 5 tests, 1 error <<< FAILURES!

[...truncated 361 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 587 suites (10 ignored), 2366 tests, 3 errors, 1 failure, 31 ignored (16 assumptions) [seed: A1D15E71442F8FE5]

Total time: 293 minutes 3 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 38 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/38/

3 tests failed.
FAILED:  org.apache.solr.cloud.CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates

Error Message:
Timeout while trying to assert number of documents @ source_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ source_collection
	at __randomizedtesting.SeedInfo.seed([C6DB3996DDD59445:15D26988984608D2]:0)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:266)
	at org.apache.solr.cloud.CdcrReplicationHandlerTest.testReplicationWithBufferedUpdates(CdcrReplicationHandlerTest.java:233)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.AssertionError: expected:<410> but was:<420>
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:257)
	... 42 more


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([C6DB3996DDD59445:4E8F064C7329F9BD]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:226)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReqParamsAPI.test

Error Message:
Could not get expected value  'CY val modified' for path 'response/params/y/c' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "response":{     "znodeVersion":0,     "params":{"x":{         "a":"A val",         "b":"B val",         "":{"v":0}}}}}

Stack Trace:
java.lang.AssertionError: Could not get expected value  'CY val modified' for path 'response/params/y/c' full output: {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "response":{
    "znodeVersion":0,
    "params":{"x":{
        "a":"A val",
        "b":"B val",
        "":{"v":0}}}}}
	at __randomizedtesting.SeedInfo.seed([C6DB3996DDD59445:4E8F064C7329F9BD]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:458)
	at org.apache.solr.handler.TestReqParamsAPI.testReqParams(TestReqParamsAPI.java:200)
	at org.apache.solr.handler.TestReqParamsAPI.test(TestReqParamsAPI.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11785 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReqParamsAPI
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/init-core-data-001
   [junit4]   2> 2190765 INFO  (SUITE-TestReqParamsAPI-seed#[C6DB3996DDD59445]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 2190765 INFO  (SUITE-TestReqParamsAPI-seed#[C6DB3996DDD59445]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 2190777 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2190779 INFO  (Thread-30614) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2190779 INFO  (Thread-30614) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2191024 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ZkTestServer start zk server on port:54070
   [junit4]   2> 2191025 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2191043 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2191055 INFO  (zkCallback-1941-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@45a39152 name:ZooKeeperConnection Watcher:127.0.0.1:54070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2191055 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2191055 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2191055 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 2191069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2191079 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2191111 INFO  (zkCallback-1942-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3c9b061a name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2191111 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2191111 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2191111 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 2191151 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 2191167 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 2191168 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 2191168 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 2191168 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 2191169 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 2191169 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 2191170 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2191170 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 2191171 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 2191171 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 2191176 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 2191176 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 2191176 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 2191176 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 2191177 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 2191177 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 2191178 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2191178 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 2191187 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2191187 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 2191188 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 2191188 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 2191189 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 2191189 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 2191265 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1
   [junit4]   2> 2191266 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 2191269 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4a58f3a4{/,null,AVAILABLE}
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4ab4befb{HTTP/1.1,[http/1.1]}{127.0.0.1:33317}
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.Server Started @2196719ms
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/tempDir-001/control/data, hostContext=/, hostPort=33317, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores}
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001'
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2191271 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2191272 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2191274 INFO  (zkCallback-1943-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@17253d20 name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2191274 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2191274 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2191276 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2191276 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/solr.xml
   [junit4]   2> 2191279 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores
   [junit4]   2> 2191281 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CoreContainer New CoreContainer 968881988
   [junit4]   2> 2191281 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001]
   [junit4]   2> 2191281 WARN  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/lib
   [junit4]   2> 2191281 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2191283 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2191283 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2191283 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2191283 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54070/solr
   [junit4]   2> 2191283 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2191284 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2191286 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2191286 INFO  (zkCallback-1946-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@75f5c9e7 name:ZooKeeperConnection Watcher:127.0.0.1:54070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2191286 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2191287 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2191291 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2191292 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@ca6bffb name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2191292 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2191293 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 2191303 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 2191304 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 2191305 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 2191311 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 2191312 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 2191313 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 2191313 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 2191314 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 2191314 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2191314 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 2191315 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 2191316 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 2191316 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2191322 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95724606305140740-127.0.0.1:33317_-n_0000000000
   [junit4]   2> 2191322 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33317_
   [junit4]   2> 2191322 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 2191323 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer Overseer (id=95724606305140740-127.0.0.1:33317_-n_0000000000) starting
   [junit4]   2> 2191324 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 2191326 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 2191327 INFO  (OverseerCollectionConfigSetProcessor-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 2191332 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33317_
   [junit4]   2> 2191332 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 2191339 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:33317_
   [junit4]   2> 2191352 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 2191367 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2191367 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 2191368 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2191368 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2191369 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores
   [junit4]   2> 2191369 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2191369 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1
   [junit4]   2> 2191369 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 2191388 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2191388 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2191389 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 2191404 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 2191404 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:33317_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 2191411 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33317",
   [junit4]   2>   "node_name":"127.0.0.1:33317_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 2191417 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33317",
   [junit4]   2>   "node_name":"127.0.0.1:33317_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2191419 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 2191422 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 2191424 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2192389 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 2192389 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 2192390 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 2192390 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 2192390 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 2192390 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1'
   [junit4]   2> 2192390 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2192390 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2192391 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 2192394 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 2192397 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 2192404 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 2192408 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 2192613 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 2192617 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 2192618 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 2192623 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2192625 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 2192626 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2192626 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 2192627 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 2192627 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 2192627 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 2192628 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data/]
   [junit4]   2> 2192628 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@68b56580
   [junit4]   2> 2192628 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data
   [junit4]   2> 2192628 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data/index/
   [junit4]   2> 2192628 WARN  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 2192628 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data/index
   [junit4]   2> 2192629 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@3277bb8c
   [junit4]   2> 2192629 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@69213e2a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@445ca31),segFN=segments_1,generation=1}
   [junit4]   2> 2192629 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 2192631 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data/
   [junit4]   2> 2192631 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/control-001/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 2192635 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 2192635 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 2192636 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 2192636 WARN  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 2192637 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2192638 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2192638 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2192644 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 2192649 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 2192649 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 2192650 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 2192650 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 2192650 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 2192650 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 2192651 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=32, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.392578125, noCFSRatio=1.0]
   [junit4]   2> 2192651 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@69213e2a lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@445ca31),segFN=segments_1,generation=1}
   [junit4]   2> 2192651 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 2192651 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2ec8d268[collection1] main]
   [junit4]   2> 2192651 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 2192652 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 2192652 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 2192652 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 2192652 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 2192652 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2192652 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 2192653 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1531593702843088896
   [junit4]   2> 2192655 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1531593702843088896
   [junit4]   2> 2192655 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 2192655 INFO  (searcherExecutor-4698-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2ec8d268[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 2192656 INFO  (coreLoadExecutor-4697-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 2192657 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:33317 collection:control_collection shard:shard1
   [junit4]   2> 2192657 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 2192659 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95724606305140740-core_node1-n_0000000000
   [junit4]   2> 2192659 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=10000
   [junit4]   2> 2192660 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 2192660 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 2192660 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33317/collection1/
   [junit4]   2> 2192660 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 2192660 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:33317/collection1/ has no replicas
   [junit4]   2> 2192661 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 2192662 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95724606305140740-core_node1-n_0000000000
   [junit4]   2> 2192668 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:33317/collection1/ shard1
   [junit4]   2> 2192670 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 2192676 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"http://127.0.0.1:33317",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 1
   [junit4]   2> 2192779 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2192821 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController We are http://127.0.0.1:33317/collection1/ and leader is http://127.0.0.1:33317/collection1/
   [junit4]   2> 2192821 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33317
   [junit4]   2> 2192821 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 2192821 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController publishing state=active
   [junit4]   2> 2192821 INFO  (coreZkRegister-4691-thread-1-processing-n:127.0.0.1:33317_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33317_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2192822 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33317",
   [junit4]   2>   "node_name":"127.0.0.1:33317_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 2
   [junit4]   2> 2192822 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "core_node_name":"core_node1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33317",
   [junit4]   2>   "node_name":"127.0.0.1:33317_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"active",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 2192908 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2192919 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2192920 INFO  (zkCallback-1949-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6380e913 name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2192920 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2192921 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2192921 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2192921 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2192922 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 2192922 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 2192922 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2192923 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2192927 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2192935 INFO  (zkCallback-1950-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@64a082ab name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2192935 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2192935 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2192935 INFO  (zkCallback-1949-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2192951 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"create",
   [junit4]   2>   "name":"collection1",
   [junit4]   2>   "numShards":"2",
   [junit4]   2>   "stateFormat":"2"} current state version: 3
   [junit4]   2> 2192951 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ClusterStateMutator building a new cName: collection1
   [junit4]   2> 2193066 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/cores/collection1
   [junit4]   2> 2193066 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001
   [junit4]   2> 2193067 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 2193068 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/collection1/state.json
   [junit4]   2> 2193068 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@226f491d{/,null,AVAILABLE}
   [junit4]   2> 2193068 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.ServerConnector Started ServerConnector@60a37698{HTTP/1.1,[http/1.1]}{127.0.0.1:49587}
   [junit4]   2> 2193068 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.e.j.s.Server Started @2198516ms
   [junit4]   2> 2193069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=49587, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/cores}
   [junit4]   2> 2193069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 2193069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001'
   [junit4]   2> 2193069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 2193069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 2193069 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2193070 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2193071 INFO  (zkCallback-1951-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@54238b69 name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2193071 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2193071 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2193072 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 2193072 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/solr.xml
   [junit4]   2> 2193086 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/cores
   [junit4]   2> 2193086 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CoreContainer New CoreContainer 1584516323
   [junit4]   2> 2193086 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001]
   [junit4]   2> 2193086 WARN  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/lib
   [junit4]   2> 2193087 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 2193089 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 2193089 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 2193089 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 2193089 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54070/solr
   [junit4]   2> 2193090 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 2193090 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2193090 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2193091 INFO  (zkCallback-1954-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@47ff0db3 name:ZooKeeperConnection Watcher:127.0.0.1:54070 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2193091 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2193091 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2193094 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2193095 INFO  (zkCallback-1955-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@369f5279 name:ZooKeeperConnection Watcher:127.0.0.1:54070/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2193095 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2193098 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 2193099 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 2193100 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 2193101 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95724606305140745-127.0.0.1:49587_-n_0000000001
   [junit4]   2> 2193101 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.LeaderElector Watching path /overseer_elect/election/95724606305140740-127.0.0.1:33317_-n_0000000000 to know if I could be the leader
   [junit4]   2> 2193101 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.ZkController Publish node=127.0.0.1:49587_ as DOWN
   [junit4]   2> 2193102 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"downnode",
   [junit4]   2>   "node_name":"127.0.0.1:49587_"} current state version: 3
   [junit4]   2> 2193102 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.NodeMutator DownNode state invoked for node: 127.0.0.1:49587_
   [junit4]   2> 2193103 INFO  (OverseerStateUpdate-95724606305140740-127.0.0.1:33317_-n_0000000000) [n:127.0.0.1:33317_    ] o.a.s.c.o.ZkStateWriter going to update_collection /collections/collection1/state.json version: 0
   [junit4]   2> 2193208 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2193209 INFO  (zkCallback-1949-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2193211 INFO  (zkCallback-1955-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2194105 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49587_
   [junit4]   2> 2194115 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:49587_
   [junit4]   2> 2194139 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2194139 INFO  (zkCallback-1949-thread-1) [    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2194139 INFO  (zkCallback-1955-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 2194147 INFO  (zkCallback-1947-thread-1-processing-n:127.0.0.1:33317_) [n:127.0.0.1:33317_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2194149 INFO  (zkCallback-1949-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2194159 INFO  (zkCallback-1955-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 2194160 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 2194160 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 2194161 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/cores
   [junit4]   2> 2194162 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=collection1, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 2194162 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.handler.TestReqParamsAPI_C6DB3996DDD59445-001/shard-1-001/cores/collection1
   [junit4]   2> 2194162 INFO  (TEST-TestReqParamsAPI.test-seed#[C6DB3996DDD59445]) [n:127.0.0.1:49587_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 2194184 INFO  (coreLoadExecutor-4708-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_ c:collection1   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 2194184 INFO  (coreLoadExecutor-4708-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_ c:collection1   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 2194185 INFO  (coreLoadExecutor-4708-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_ c:collection1   x:collection1] o.a.s.c.ZkController Registering watch for collection collection1
   [junit4]   2> 2194185 INFO  (coreLoadExecutor-4708-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader addZkWatch [collection1]
   [junit4]   2> 2194185 INFO  (coreLoadExecutor-4708-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.1:49587_ c:collection1   x:collection1] o.a.s.c.c.ZkStateReader Add data for [collection1] ver [1]
   [junit4]   2> 2194185 INFO  (coreLoadExecutor-4708-thread-1-processing-n:127.0.0.1:49587_) [n:127.0.0.

[...truncated too long message...]

72)
   [junit4]   2> 	at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.shutdown(DefaultMetricsSystem.java:68)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics.shutdown(NameNodeMetrics.java:145)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.NameNode.stop(NameNode.java:822)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1720)
   [junit4]   2> 	at org.apache.hadoop.hdfs.MiniDFSCluster.shutdown(MiniDFSCluster.java:1699)
   [junit4]   2> 	at org.apache.solr.cloud.hdfs.HdfsTestUtil.teardownClass(HdfsTestUtil.java:198)
   [junit4]   2> 	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.hdfsFailoverAfterClass(SharedFSAutoReplicaFailoverTest.java:82)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 5530464 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[C6DB3996DDD59445]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_C6DB3996DDD59445-001
   [junit4]   2> Apr 14, 2016 2:41:55 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60), sim=ClassicSimilarity, locale=ar-LB, timezone=Antarctica/Macquarie
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=130663280,total=504889344
   [junit4]   2> NOTE: All tests run in this JVM: [DocValuesMultiTest, TestIndexSearcher, ReplicationFactorTest, CollectionStateFormat2Test, TestCollapseQParserPlugin, TestCloudDeleteByQuery, SimpleMLTQParserTest, XmlUpdateRequestHandlerTest, SuggesterTSTTest, VMParamsZkACLAndCredentialsProvidersTest, TestNoOpRegenerator, TestPHPSerializedResponseWriter, HdfsChaosMonkeyNothingIsSafeTest, TestReload, BadIndexSchemaTest, TestBlobHandler, HdfsTlogReplayBufferedWhileIndexingTest, OverseerCollectionConfigSetProcessorTest, TestConfigSets, NoCacheHeaderTest, OverseerTaskQueueTest, TestSolrConfigHandlerCloud, BinaryUpdateRequestHandlerTest, TestSuggestSpellingConverter, ConnectionManagerTest, FullSolrCloudDistribCmdsTest, CircularListTest, TestSimpleQParserPlugin, BaseCdcrDistributedZkTest, CleanupOldIndexTest, TestExtendedDismaxParser, SolrCoreCheckLockOnStartupTest, TestWriterPerf, DebugComponentTest, SimpleFacetsTest, UUIDUpdateProcessorFallbackTest, TestSolrCloudWithKerberosAlt, BigEndianAscendingWordDeserializerTest, FieldAnalysisRequestHandlerTest, TestInitQParser, TestTrieFacet, TestManagedResourceStorage, DistributedFacetPivotSmallAdvancedTest, TestExactSharedStatsCache, TestSolrFieldCacheMBean, ForceLeaderTest, LeaderInitiatedRecoveryOnCommitTest, SolrTestCaseJ4Test, CoreAdminHandlerTest, PreAnalyzedUpdateProcessorTest, CursorMarkTest, TestManagedSchema, ExitableDirectoryReaderTest, TestSolrJ, TestFuzzyAnalyzedSuggestions, BasicAuthIntegrationTest, SSLMigrationTest, TestMacros, TestRTimerTree, TestCoreDiscovery, ZkStateWriterTest, SolrCloudExampleTest, RulesTest, TestLeaderInitiatedRecoveryThread, TestRecoveryHdfs, FacetPivotSmallTest, TestCSVResponseWriter, TestScoreJoinQPScore, TestTolerantUpdateProcessorRandomCloud, TestFastWriter, AssignTest, TestSweetSpotSimilarityFactory, TestTolerantSearch, CoreAdminRequestStatusTest, DataDrivenBlockJoinTest, TestSchemaVersionResource, BlockJoinFacetSimpleTest, HighlighterConfigTest, ScriptEngineTest, ExplicitHLLTest, PrimitiveFieldTypeTest, TestConfigSetsAPIZkFailure, AtomicUpdatesTest, TestHighFrequencyDictionaryFactory, CurrencyFieldXmlFileTest, TestSearchPerf, TestStressReorder, StressHdfsTest, TestSolrDynamicMBean, TestReqParamsAPI, BJQParserTest, TestManagedSynonymFilterFactory, DocExpirationUpdateProcessorFactoryTest, DistribJoinFromCollectionTest, TestFieldSortValues, TestCollectionAPI, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest, OpenCloseCoreStressTest, OverseerTest, BasicZkTest, RecoveryZkTest, TestRandomFaceting, AutoCommitTest, TestDistributedGrouping, TestRealTimeGet, HardAutoCommitTest, DistributedTermsComponentTest, StatsComponentTest, SolrCmdDistributorTest, QueryElevationComponentTest, TestSort, ShowFileRequestHandlerTest, SolrIndexSplitterTest, SpellCheckCollatorTest, TestFoldingMultitermQuery, SuggesterTest, SpatialFilterTest, PolyFieldTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, DirectUpdateHandlerOptimizeTest, StatelessScriptUpdateProcessorFactoryTest, TestSurroundQueryParser, TestQueryUtils, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, TestQueryTypes, TermsComponentTest, TestValueSourceCache, TermVectorComponentTest, RequiredFieldsTest, FastVectorHighlighterTest, RegexBoostProcessorTest, TestJmxIntegration, ReturnFieldsTest, QueryParsingTest, UniqFieldsUpdateProcessorFactoryTest, PingRequestHandlerTest, SOLR749Test, TestQuerySenderListener, UpdateParamsTest, AlternateDirectoryTest, TestSolrIndexConfig, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, ConnectionReuseTest, CdcrReplicationHandlerTest, CustomCollectionTest, DistributedVersionInfoTest, LeaderFailoverAfterPartitionTest, MultiThreadedOCPTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest]
   [junit4] Completed [440/586 (3!)] on J2 in 176.76s, 1 test, 1 failure <<< FAILURES!

[...truncated 469 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 586 suites (10 ignored), 2364 tests, 3 failures, 31 ignored (16 assumptions) [seed: C6DB3996DDD59445]

Total time: 312 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 37 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/37/

6 tests failed.
FAILED:  org.apache.solr.client.solrj.ConnectionReuseTest.test

Error Message:
We expected all communication via streaming client to use one connection! expected=283 got=280

Stack Trace:
java.lang.AssertionError: We expected all communication via streaming client to use one connection! expected=283 got=280
	at __randomizedtesting.SeedInfo.seed([DCE3894FBD1E9D4A:54B7B69513E2F0B2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.client.solrj.ConnectionReuseTest.test(ConnectionReuseTest.java:150)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.test

Error Message:
Exactly one shard should have changed, instead: [shard2, shard1] nodes=([core_node3(shard2), core_node2(shard1), core_node4(shard1)]) expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: Exactly one shard should have changed, instead: [shard2, shard1] nodes=([core_node3(shard2), core_node2(shard1), core_node4(shard1)]) expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([DCE3894FBD1E9D4A:54B7B69513E2F0B2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.DistribDocExpirationUpdateProcessorTest.test(DistribDocExpirationUpdateProcessorTest.java:118)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=17450, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=17450, name=collection3, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:35990: collection already exists: awholynewstresscollection_collection3_0
	at __randomizedtesting.SeedInfo.seed([DCE3894FBD1E9D4A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1593)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1614)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([DCE3894FBD1E9D4A:B7AC2932C4114070]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 7 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 7 object(s) that were not released!!! [NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory, NRTCachingDirectory]
	at __randomizedtesting.SeedInfo.seed([DCE3894FBD1E9D4A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:256)
	at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
replicaCount expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: replicaCount expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([DCE3894FBD1E9D4A:54B7B69513E2F0B2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoConfigSetExist(CollectionsAPIDistributedZkTest.java:601)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11486 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/init-core-data-001
   [junit4]   2> 1483918 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[DCE3894FBD1E9D4A]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false)
   [junit4]   2> 1483918 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[DCE3894FBD1E9D4A]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /k/yk
   [junit4]   2> 1483919 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1483919 INFO  (Thread-2917) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1483919 INFO  (Thread-2917) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1484020 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.ZkTestServer start zk server on port:54931
   [junit4]   2> 1484020 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484026 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484035 INFO  (zkCallback-1209-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@618e9edf name:ZooKeeperConnection Watcher:127.0.0.1:54931 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484035 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484035 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484035 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1484038 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1540fcdd7010000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1484039 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484050 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484058 INFO  (zkCallback-1210-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@29119708 name:ZooKeeperConnection Watcher:127.0.0.1:54931/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484058 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484059 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484059 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1484059 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1484060 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1484061 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1484061 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1484061 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1484062 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1484062 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1484063 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1484063 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1484064 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1484064 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1484065 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1484065 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1484065 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1484066 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1484066 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1484066 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1484067 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1484067 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1484068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1484068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1484068 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1484069 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1484074 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1484074 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1484086 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484095 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484103 INFO  (zkCallback-1211-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@488b5f03 name:ZooKeeperConnection Watcher:127.0.0.1:54931/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484103 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484104 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484104 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484114 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484141 INFO  (zkCallback-1212-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6cd7970e name:ZooKeeperConnection Watcher:127.0.0.1:54931 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484141 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484141 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484141 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1484143 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484166 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484176 INFO  (zkCallback-1213-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3671f139 name:ZooKeeperConnection Watcher:127.0.0.1:54931/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484176 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484176 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484176 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 1484176 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 1484177 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 1484177 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 1484178 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 1484178 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 1484179 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1484179 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1484180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 1484180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 1484180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 1484180 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 1484181 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 1484181 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 1484182 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1484182 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 1484183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1484183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1484183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 1484183 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 1484184 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 1484184 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 1484185 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 1484185 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 1484367 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1
   [junit4]   2> 1484382 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1484479 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5516e0c9{/k/yk,null,AVAILABLE}
   [junit4]   2> 1484480 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.e.j.u.s.SslContextFactory x509=X509@58068d88(solrtest,h=[],w=[]) for SslContextFactory@478985fb(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 1484495 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.e.j.s.ServerConnector Started ServerConnector@2b8f2a4f{SSL,[ssl, http/1.1]}{127.0.0.1:49964}
   [junit4]   2> 1484495 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.e.j.s.Server Started @1490483ms
   [junit4]   2> 1484518 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/tempDir-001/control/data, hostContext=/k/yk, hostPort=49964, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores}
   [junit4]   2> 1484518 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1484518 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001'
   [junit4]   2> 1484518 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1484518 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1484518 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484523 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484540 INFO  (zkCallback-1214-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@73a26ced name:ZooKeeperConnection Watcher:127.0.0.1:54931/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484540 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484540 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484567 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1484567 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/solr.xml
   [junit4]   2> 1484571 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores
   [junit4]   2> 1484571 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.CoreContainer New CoreContainer 1392767
   [junit4]   2> 1484571 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001]
   [junit4]   2> 1484571 WARN  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/lib
   [junit4]   2> 1484572 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1484575 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1484576 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1484576 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1484576 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54931/solr
   [junit4]   2> 1484576 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1484576 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1484841 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484842 INFO  (zkCallback-1217-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@17f0cacd name:ZooKeeperConnection Watcher:127.0.0.1:54931 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484844 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484845 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1484846 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x1540fcdd7010006, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 1484850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1484850 INFO  (zkCallback-1218-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c4d1fe name:ZooKeeperConnection Watcher:127.0.0.1:54931/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1484850 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1484852 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1484859 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1484868 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1484870 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1484871 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1484874 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1484876 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1484876 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1484878 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1484880 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1484883 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1484886 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1484887 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1484889 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1484890 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000
   [junit4]   2> 1484891 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49964_k%2Fyk
   [junit4]   2> 1484891 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1484892 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.Overseer Overseer (id=95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) starting
   [junit4]   2> 1484894 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1484911 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1484918 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1484919 INFO  (OverseerCollectionConfigSetProcessor-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1485003 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49964_k%2Fyk
   [junit4]   2> 1485004 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1485006 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:49964_k%2Fyk
   [junit4]   2> 1485019 INFO  (zkCallback-1218-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1485019 INFO  (zkCallback-1218-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1485041 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1485041 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1485047 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores
   [junit4]   2> 1485056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1485056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1
   [junit4]   2> 1485056 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1485063 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1485063 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1485071 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1485071 INFO  (TEST-CollectionsAPIDistributedZkTest.test-seed#[DCE3894FBD1E9D4A]) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1485071 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1485071 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:49964/k/yk",
   [junit4]   2>   "node_name":"127.0.0.1:49964_k%2Fyk",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1485072 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"https://127.0.0.1:49964/k/yk",
   [junit4]   2>   "node_name":"127.0.0.1:49964_k%2Fyk",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1485072 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1485072 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1485073 INFO  (zkCallback-1218-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1486071 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1486071 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1486074 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1486074 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1486075 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1486075 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1'
   [junit4]   2> 1486075 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1486075 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1486076 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1486123 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1486126 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1486145 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1486166 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1486443 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 1486451 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1486452 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1486462 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1486464 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1486475 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1486475 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 1486476 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1486476 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1486477 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 1486477 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1/data/]
   [junit4]   2> 1486477 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@614ac299
   [junit4]   2> 1486477 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1/data
   [junit4]   2> 1486482 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1/data/index/
   [junit4]   2> 1486482 WARN  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1486484 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1/data/index
   [junit4]   2> 1486485 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=32, maxMergedSegmentMB=29.4736328125, floorSegmentMB=0.970703125, forceMergeDeletesPctAllowed=11.990682822227466, segmentsPerTier=38.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1486485 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@4c1796d9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11d1a942),segFN=segments_1,generation=1}
   [junit4]   2> 1486485 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1486516 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1486517 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1486517 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1486517 WARN  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1486518 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1486531 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1486531 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1486532 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1486543 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1486543 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1486554 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_DCE3894FBD1E9D4A-001/control-001/cores/collection1/data/
   [junit4]   2> 1486554 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1486555 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1486562 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1486563 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1486718 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=45, maxMergeAtOnceExplicit=37, maxMergedSegmentMB=35.5498046875, floorSegmentMB=1.384765625, forceMergeDeletesPctAllowed=4.059690790772287, segmentsPerTier=18.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0
   [junit4]   2> 1486749 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@4c1796d9 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11d1a942),segFN=segments_1,generation=1}
   [junit4]   2> 1486749 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1486749 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7121a37a[collection1] main]
   [junit4]   2> 1486749 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1486750 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1486750 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1486750 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1486750 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1486750 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1486751 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1531501904272556032
   [junit4]   2> 1486753 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1531501904272556032
   [junit4]   2> 1486753 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1486754 INFO  (coreLoadExecutor-2988-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1486758 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:https://127.0.0.1:49964/k/yk collection:control_collection shard:shard1
   [junit4]   2> 1486759 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1486761 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95718868831502343-core_node1-n_0000000000
   [junit4]   2> 1486761 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 1486762 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection"} current state version: 1
   [junit4]   2> 1486762 INFO  (zkCallback-1218-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1486764 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1486764 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1486764 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:49964/k/yk/collection1/
   [junit4]   2> 1486764 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1486764 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy https://127.0.0.1:49964/k/yk/collection1/ has no replicas
   [junit4]   2> 1486764 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 1486765 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/95718868831502343-core_node1-n_0000000000
   [junit4]   2> 1486766 INFO  (OverseerStateUpdate-95718868831502343-127.0.0.1:49964_k%2Fyk-n_0000000000) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "operation":"leader",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "base_url":"https://127.0.0.1:49964/k/yk",
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "state":"active"} current state version: 2
   [junit4]   2> 1486767 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:49964/k/yk/collection1/ shard1
   [junit4]   2> 1486844 INFO  (searcherExecutor-2989-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:49964_k%2Fyk c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7121a37a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1486868 INFO  (zkCallback-1218-thread-1-processing-n:127.0.0.1:49964_k%2Fyk) [n:127.0.0.1:49964_k%2Fyk    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1486918 INFO  (coreZkRegister-2982-thread-1-processing-n:127.0.0.1:49964_k%2Fyk x:collection1 s:shard1 c:cont

[...truncated too long message...]

loud.ConnectionManager@5a30ea82 name:ZooKeeperConnection Watcher:127.0.0.1:44487 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5787997 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5787998 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 5787998 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 5788039 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 5788045 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 5788056 INFO  (zkCallback-4394-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3ecb9dc5 name:ZooKeeperConnection Watcher:127.0.0.1:44487/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 5788056 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5788057 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 5788057 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 5788058 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 5788058 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 5788059 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 5788060 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 5788061 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 5788061 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 5788061 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 5788062 INFO  (zkCallback-4394-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 5788063 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 5788063 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 5788064 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 5788064 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.c.ZkStateReader Removing watch for uninteresting collection [c1]
   [junit4]   2> 5788066 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44487 44487
   [junit4]   2> 5788258 INFO  (Thread-28649) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:44487 44487
   [junit4]   2> 5788259 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[DCE3894FBD1E9D4A]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalCollectionWatchedNotWatched
   [junit4]   2> 5788260 INFO  (SUITE-ZkStateReaderTest-seed#[DCE3894FBD1E9D4A]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_DCE3894FBD1E9D4A-001
   [junit4]   2> Apr 13, 2016 2:38:53 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Lucene60, sim=ClassicSimilarity, locale=ar-IQ, timezone=ACT
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=193877440,total=491782144
   [junit4]   2> NOTE: All tests run in this JVM: [DocumentAnalysisRequestHandlerTest, TestMiniSolrCloudCluster, DistributedSuggestComponentTest, TestCryptoKeys, CdcrReplicationDistributedZkTest, CustomCollectionTest, CloneFieldUpdateProcessorFactoryTest, TestLuceneMatchVersion, TestStressUserVersions, RecoveryAfterSoftCommitTest, TestRawTransformer, RestartWhileUpdatingTest, TestAuthorizationFramework, TestBadConfig, TestFreeTextSuggestions, TestAnalyzeInfixSuggestions, TestCollationFieldDocValues, TestTestInjection, CoreAdminCreateDiscoverTest, TestDistribIDF, TestRangeQuery, AnalysisAfterCoreReloadTest, TestDefaultSearchFieldResource, OutputWriterTest, CdcrVersionReplicationTest, TestAtomicUpdateErrorCases, TestPhraseSuggestions, ConcurrentDeleteAndCreateCollectionTest, TestMergePolicyConfig, TestRandomDVFaceting, InfoHandlerTest, TestSearcherReuse, TestRandomFaceting, AlternateDirectoryTest, SignatureUpdateProcessorFactoryTest, DistributedQueryElevationComponentTest, TestCustomDocTransformer, ChangedSchemaMergeTest, EchoParamsTest, TestReplicationHandler, TestRTGBase, QueryEqualityTest, TestDocSet, CacheHeaderTest, OverseerRolesTest, DirectoryFactoryTest, TestCloudManagedSchema, TestQueryWrapperFilter, DistributedSpellCheckComponentTest, TestHighlightDedupGrouping, PingRequestHandlerTest, FileBasedSpellCheckerTest, DistributedFacetPivotWhiteBoxTest, TestConfig, HdfsThreadLeakTest, TestUseDocValuesAsStored, PrimUtilsTest, SaslZkACLProviderTest, TestBinaryField, TestValueSourceCache, TolerantUpdateProcessorTest, TestConfigSetProperties, TestJoin, TestManagedResource, EnumFieldTest, DistributedQueueTest, TestSortByMinMaxFunction, TestPseudoReturnFields, ParsingFieldUpdateProcessorsTest, RequiredFieldsTest, PreAnalyzedFieldTest, TestRealTimeGet, BlockJoinFacetRandomTest, QueryElevationComponentTest, DistributedFacetPivotLongTailTest, DisMaxRequestHandlerTest, TestOmitPositions, TestSolrConfigHandler, TestExceedMaxTermLength, URLClassifyProcessorTest, TestManagedStopFilterFactory, BlockDirectoryTest, LeaderElectionIntegrationTest, TestScoreJoinQPNoScore, TestSchemaSimilarityResource, HdfsCollectionsAPIDistributedZkTest, ZkNodePropsTest, PathHierarchyTokenizerFactoryTest, TestShortCircuitedRequests, SyncSliceTest, DefaultValueUpdateProcessorTest, HdfsLockFactoryTest, AsyncCallRequestStatusResponseTest, IndexSchemaTest, PeerSyncTest, SolrXmlInZkTest, TestTolerantUpdateProcessorCloud, SolrTestCaseJ4Test, TestCrossCoreJoin, TestCursorMarkWithoutUniqueKey, ConnectionReuseTest, CollectionReloadTest, CollectionTooManyReplicasTest, DeleteLastCustomShardedReplicaTest, DeleteReplicaTest, DistribCursorPagingTest, LeaderInitiatedRecoveryOnShardRestartTest, MultiThreadedOCPTest, TestCollectionAPI, TestConfigSetsAPI, TestDistribDocBasedVersion, TestLeaderElectionZkExpiry, TestLeaderInitiatedRecoveryThread, TestReplicaProperties, TriLevelCompositeIdRoutingTest, HdfsChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, HdfsNNFailoverTest, HdfsRecoveryZkTest, HdfsSyncSliceTest, TestClusterStateMutator, ZkStateReaderTest]
   [junit4] Completed [476/585 (6!)] on J0 in 3.59s, 5 tests, 1 error <<< FAILURES!

[...truncated 340 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2363 tests, 1 suite-level error, 2 errors, 3 failures, 31 ignored (16 assumptions) [seed: DCE3894FBD1E9D4A]

Total time: 293 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 36 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/36/

5 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Captured an uncaught exception in thread: Thread[id=48119, name=Thread-47818, state=RUNNABLE, group=TGRP-TestDistributedSearch]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=48119, name=Thread-47818, state=RUNNABLE, group=TGRP-TestDistributedSearch]
	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02:8A3E15C1D1C6C6FA]:0)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:48767//collection1: java.lang.NullPointerException
	at org.apache.solr.search.grouping.distributed.responseprocessor.TopGroupsShardResponseProcessor.process(TopGroupsShardResponseProcessor.java:105)
	at org.apache.solr.handler.component.QueryComponent.handleGroupedResponses(QueryComponent.java:753)
	at org.apache.solr.handler.component.QueryComponent.handleResponses(QueryComponent.java:736)
	at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:426)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:155)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2018)
	at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:652)
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:460)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:229)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:184)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1676)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:462)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
	at org.eclipse.jetty.server.Server.handle(Server.java:518)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:186)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
	at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
	at java.lang.Thread.run(Thread.java:745)

	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.TestDistributedSearch$1.run(TestDistributedSearch.java:1114)


FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.BasicDistributedZkTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02]:0)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_26A2A1B7F3AAB02-001/solr-instance-017/./collection1/data/index.20160413001924298, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_26A2A1B7F3AAB02-001/solr-instance-017/./collection1/data/index.20160413001924113, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_26A2A1B7F3AAB02-001/solr-instance-017/./collection1/data] expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_26A2A1B7F3AAB02-001/solr-instance-017/./collection1/data/index.20160413001924298, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_26A2A1B7F3AAB02-001/solr-instance-017/./collection1/data/index.20160413001924113, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_26A2A1B7F3AAB02-001/solr-instance-017/./collection1/data] expected:<2> but was:<3>
	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02:F519C443B9D204E4]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.handler.TestReplicationHandler.checkForSingleIndex(TestReplicationHandler.java:819)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1249)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
replicaCount expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: replicaCount expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02:8A3E15C1D1C6C6FA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoConfigSetExist(CollectionsAPIDistributedZkTest.java:601)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11484 lines...]
   [junit4] Suite: org.apache.solr.TestDistributedSearch
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/init-core-data-001
   [junit4]   2> 110244 INFO  (SUITE-TestDistributedSearch-seed#[26A2A1B7F3AAB02]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 110244 INFO  (SUITE-TestDistributedSearch-seed#[26A2A1B7F3AAB02]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 110695 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1
   [junit4]   2> 110792 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.u.log Logging initialized @117063ms
   [junit4]   2> 111338 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 111665 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@353d6714{/,null,AVAILABLE}
   [junit4]   2> 111770 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.u.s.SslContextFactory x509=X509@33be9386(solrtest,h=[],w=[]) for SslContextFactory@6d2b7e9a(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 112712 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.ServerConnector Started ServerConnector@79c9459c{SSL,[ssl, http/1.1]}{127.0.0.1:39113}
   [junit4]   2> 112713 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.Server Started @118997ms
   [junit4]   2> 112713 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=39113, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores}
   [junit4]   2> 112791 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 112792 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control'
   [junit4]   2> 112792 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 112792 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 112794 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/solr.xml
   [junit4]   2> 112861 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores
   [junit4]   2> 112861 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer New CoreContainer 1767957393
   [junit4]   2> 112861 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control]
   [junit4]   2> 112861 WARN  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/lib
   [junit4]   2> 112862 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 112869 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 112871 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 112871 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 112872 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 112872 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 112872 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores
   [junit4]   2> 112874 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/, configSet=collection1}
   [junit4]   2> 112874 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1
   [junit4]   2> 112875 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 112888 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 112897 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 112909 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 112910 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 112910 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 112911 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 112911 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 112989 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 113006 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 113054 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 113102 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 113966 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 114046 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 114056 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 114101 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 114103 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 114105 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 114106 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 114123 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 114124 INFO  (coreLoadExecutor-31-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1
   [junit4]   2> 114124 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 114124 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data/]
   [junit4]   2> 114139 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap No JMX servers found, not exposing Solr information with JMX.
   [junit4]   2> 114140 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data
   [junit4]   2> 114140 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data/index/
   [junit4]   2> 114140 WARN  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 114140 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data/index
   [junit4]   2> 114141 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=20.9775390625, floorSegmentMB=0.8330078125, forceMergeDeletesPctAllowed=26.40655049622171, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.22775814159314997
   [junit4]   2> 114145 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@78588926 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@76a5e1f3),segFN=segments_1,generation=1}
   [junit4]   2> 114145 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 114152 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 114152 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 114152 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 114152 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 114153 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 114155 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 114156 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 114156 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 114157 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 114158 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data/
   [junit4]   2> 114167 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/control/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 114172 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 114175 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,spellCheckCompRH,lazy,/update/json/docs,/admin/luke,/export,spellCheckWithWordbreak_Direct,tvrh,/get,/admin/properties,/update/json,mltrh,/admin/threads,/search-facet-def,/update/csv,/sql,/search-facet-invariants,/admin/segments,spellCheckCompRH1,/mlt,/admin/system,/replication,dismax,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,spellCheckWithWordbreak,mock,/admin/ping,/update,/admin/file,/terms,spellCheckCompRH_Direct,/debug/dump
   [junit4]   2> 114175 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 114205 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 114205 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 114205 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=94.158203125, floorSegmentMB=1.8671875, forceMergeDeletesPctAllowed=22.9009354229174, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2977926661029187
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@78588926 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@76a5e1f3),segFN=segments_1,generation=1}
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7b6e4a1[collection1] main]
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 114207 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 114240 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 114313 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 114313 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 114313 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 114313 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 114313 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 114314 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 114314 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 114314 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 114314 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 114315 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 114315 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7b6e4a1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 114318 INFO  (coreLoadExecutor-31-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 114935 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1
   [junit4]   2> 114942 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 114976 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@554221a4{/,null,AVAILABLE}
   [junit4]   2> 114977 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.u.s.SslContextFactory x509=X509@4ce2f8aa(solrtest,h=[],w=[]) for SslContextFactory@7c950be3(file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/etc/test/solrtest.keystore,file:///x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/server/etc/test/solrtest.keystore)
   [junit4]   2> 114998 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.ServerConnector Started ServerConnector@110abc34{SSL,[ssl, http/1.1]}{127.0.0.1:53068}
   [junit4]   2> 114998 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.e.j.s.Server Started @121283ms
   [junit4]   2> 114998 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/, hostPort=53068, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores}
   [junit4]   2> 114999 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 114999 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0'
   [junit4]   2> 114999 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 114999 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 114999 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/solr.xml
   [junit4]   2> 115019 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores
   [junit4]   2> 115019 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer New CoreContainer 1754911478
   [junit4]   2> 115019 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0]
   [junit4]   2> 115019 WARN  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/lib
   [junit4]   2> 115019 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : https,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 115021 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 115023 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 115023 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 115046 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 115046 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 115056 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores
   [junit4]   2> 115072 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/, configSet=collection1}
   [junit4]   2> 115072 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1
   [junit4]   2> 115072 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 115099 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 115099 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 115114 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 115114 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 115114 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 115115 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/lib/classes/' to classloader
   [junit4]   2> 115115 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrResourceLoader Adding 'file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/lib/README' to classloader
   [junit4]   2> 115251 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 115293 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 115588 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 115613 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 115807 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 115835 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 115837 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 115882 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 116148 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 116150 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 116151 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 116153 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1'
   [junit4]   2> 116153 INFO  (coreLoadExecutor-41-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1
   [junit4]   2> 116153 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 116153 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data/]
   [junit4]   2> 116153 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.JmxMonitoredMap No JMX servers found, not exposing Solr information with JMX.
   [junit4]   2> 116161 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data
   [junit4]   2> 116161 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data/index/
   [junit4]   2> 116161 WARN  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 116162 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data/index
   [junit4]   2> 116162 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=39, maxMergeAtOnceExplicit=24, maxMergedSegmentMB=20.9775390625, floorSegmentMB=0.8330078125, forceMergeDeletesPctAllowed=26.40655049622171, segmentsPerTier=14.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.22775814159314997
   [junit4]   2> 116175 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@55c4da5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@154bccf0),segFN=segments_1,generation=1}
   [junit4]   2> 116175 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 116187 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 116195 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 116195 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 116196 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe-allfields"
   [junit4]   2> 116196 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 116196 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 116197 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 116197 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "uniq-fields"
   [junit4]   2> 116198 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 116198 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 116198 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 116198 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 116202 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 116204 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 116204 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data/
   [junit4]   2> 116206 WARN  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.DirectoryFactory /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.TestDistributedSearch_26A2A1B7F3AAB02-001/tempDir-001/shard0/cores/collection1/data/ does not point to a valid data directory; skipping clean-up of old index directories.
   [junit4]   2> 116208 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 116209 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 116219 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 116227 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,spellCheckCompRH,lazy,/update/json/docs,/admin/luke,/export,spellCheckWithWordbreak_Direct,tvrh,/get,/admin/properties,/update/json,mltrh,/admin/threads,/search-facet-def,/update/csv,/sql,/search-facet-invariants,/admin/segments,spellCheckCompRH1,/mlt,/admin/system,/replication,dismax,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,spellCheckWithWordbreak,mock,/admin/ping,/update,/admin/file,/terms,spellCheckCompRH_Direct,/debug/dump
   [junit4]   2> 116227 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 116233 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 116233 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 116233 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=3, maxMergeAtOnceExplicit=5, maxMergedSegmentMB=94.158203125, floorSegmentMB=1.8671875, forceMergeDeletesPctAllowed=22.9009354229174, segmentsPerTier=21.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.2977926661029187
   [junit4]   2> 116233 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@55c4da5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@154bccf0),segFN=segments_1,generation=1}
   [junit4]   2> 116233 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 116243 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@725d1bef[collection1] main]
   [junit4]   2> 116244 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageIO=org.apache.solr.rest.ManagedResourceStorage$InMemoryStorageIO}
   [junit4]   2> 116244 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using InMemoryStorage
   [junit4]   2> 116245 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 116246 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 116272 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=direct,classname=DirectSolrSpellChecker,field=lowerfilt,minQueryLength=3}
   [junit4]   2> 116371 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 116372 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 116373 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: direct
   [junit4]   2> 116373 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
   [junit4]   2> 116373 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: multipleFields
   [junit4]   2> 116373 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: jarowinkler
   [junit4]   2> 116373 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: external
   [junit4]   2> 116373 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: freq
   [junit4]   2> 116374 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: fqcn
   [junit4]   2> 116374 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: perDict
   [junit4]   2> 116374 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@725d1bef[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 116487 INFO  (coreLoadExecutor-41-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 116610 INFO  (TEST-TestDistributedSearch.test-seed#[26A2A1B7F3AAB02]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 117831 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 117832 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 36
   [junit4]   2> 118125 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.DirectUpdateHandler2 [collection1] REMOVING ALL DOCUMENTS FROM INDEX
   [junit4]   2> 118143 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{deleteByQuery=*:*} 0 19
   [junit4]   2> 118210 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1]} 0 47
   [junit4]   2> 118229 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1]} 0 14
   [junit4]   2> 118244 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[2]} 0 9
   [junit4]   2> 118269 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[2]} 0 11
   [junit4]   2> 118274 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[3]} 0 1
   [junit4]   2> 118279 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[3]} 0 1
   [junit4]   2> 118290 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[4]} 0 1
   [junit4]   2> 118295 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[4]} 0 1
   [junit4]   2> 118300 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[5]} 0 1
   [junit4]   2> 118304 INFO  (qtp865475013-63) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[5]} 0 1
   [junit4]   2> 118308 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[6]} 0 1
   [junit4]   2> 118313 INFO  (qtp865475013-63) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[6]} 0 0
   [junit4]   2> 118317 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[7]} 0 1
   [junit4]   2> 118331 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[7]} 0 1
   [junit4]   2> 118369 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[8]} 0 1
   [junit4]   2> 118380 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[8]} 0 1
   [junit4]   2> 118384 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[9]} 0 1
   [junit4]   2> 118389 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[9]} 0 1
   [junit4]   2> 118600 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 118761 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@78588926 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@76a5e1f3),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@78588926 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@76a5e1f3),segFN=segments_2,generation=2}
   [junit4]   2> 118761 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 119367 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@1982da40[collection1] main]
   [junit4]   2> 119367 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 119367 INFO  (searcherExecutor-32-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@1982da40[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.1.0):c9)))}
   [junit4]   2> 119379 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 779
   [junit4]   2> 119629 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 119712 INFO  (qtp865475013-59) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@55c4da5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@154bccf0),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@55c4da5e lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@154bccf0),segFN=segments_2,generation=2}
   [junit4]   2> 119713 INFO  (qtp865475013-59) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 119755 INFO  (qtp865475013-59) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2c616591[collection1] main]
   [junit4]   2> 119756 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 119756 INFO  (searcherExecutor-42-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@2c616591[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.1.0):c9)))}
   [junit4]   2> 119759 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={waitSearcher=true&commit=true&softCommit=false&wt=javabin&version=2}{commit=} 0 376
   [junit4]   2> 119766 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[10]} 0 2
   [junit4]   2> 119781 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[10]} 0 7
   [junit4]   2> 119786 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[11]} 0 1
   [junit4]   2> 119791 INFO  (qtp865475013-63) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[11]} 0 1
   [junit4]   2> 119795 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[12]} 0 1
   [junit4]   2> 119800 INFO  (qtp865475013-63) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[12]} 0 1
   [junit4]   2> 119805 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[13]} 0 1
   [junit4]   2> 119809 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[13]} 0 1
   [junit4]   2> 119819 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1001]} 0 2
   [junit4]   2> 119826 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[1001]} 0 1
   [junit4]   2> 119838 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[14]} 0 1
   [junit4]   2> 119854 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[14]} 0 1
   [junit4]   2> 119869 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[15]} 0 1
   [junit4]   2> 119883 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[15]} 0 1
   [junit4]   2> 120544 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[16]} 0 1
   [junit4]   2> 120556 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[16]} 0 0
   [junit4]   2> 120561 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[17]} 0 1
   [junit4]   2> 120569 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[17]} 0 1
   [junit4]   2> 120575 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 120579 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[100]} 0 0
   [junit4]   2> 120585 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 120588 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[101]} 0 0
   [junit4]   2> 120593 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 120596 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[102]} 0 0
   [junit4]   2> 120605 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 120608 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[103]} 0 0
   [junit4]   2> 120611 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 120627 INFO  (qtp865475013-63) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[104]} 0 0
   [junit4]   2> 120632 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 120640 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[105]} 0 0
   [junit4]   2> 120649 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 120667 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[106]} 0 0
   [junit4]   2> 120671 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 120674 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[107]} 0 0
   [junit4]   2> 120687 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[108]} 0 7
   [junit4]   2> 120691 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[108]} 0 0
   [junit4]   2> 120713 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 120716 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[109]} 0 0
   [junit4]   2> 120720 INFO  (qtp1401126775-45) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 120723 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[110]} 0 0
   [junit4]   2> 120727 INFO  (qtp1401126775-45) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 120730 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[111]} 0 0
   [junit4]   2> 120740 INFO  (qtp1401126775-45) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[112]} 0 6
   [junit4]   2> 120744 INFO  (qtp865475013-65) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[112]} 0 0
   [junit4]   2> 120747 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 120750 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[113]} 0 0
   [junit4]   2> 120754 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 120759 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[114]} 0 0
   [junit4]   2> 120764 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 120771 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[115]} 0 0
   [junit4]   2> 120778 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 120789 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[116]} 0 0
   [junit4]   2> 120797 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 120805 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[117]} 0 0
   [junit4]   2> 120808 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 120811 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[118]} 0 0
   [junit4]   2> 120815 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 120818 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[119]} 0 0
   [junit4]   2> 120825 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[120]} 0 0
   [junit4]   2> 120859 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[120]} 0 17
   [junit4]   2> 120864 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 120877 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[121]} 0 0
   [junit4]   2> 120880 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[122]} 0 0
   [junit4]   2> 121038 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[122]} 0 1
   [junit4]   2> 121047 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 121050 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[123]} 0 0
   [junit4]   2> 121054 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 121057 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[124]} 0 0
   [junit4]   2> 121060 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 121063 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[125]} 0 0
   [junit4]   2> 121066 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 121069 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[126]} 0 0
   [junit4]   2> 121072 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 121076 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[127]} 0 0
   [junit4]   2> 121079 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 121083 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[128]} 0 0
   [junit4]   2> 121086 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 121090 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[129]} 0 0
   [junit4]   2> 121093 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 121098 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[130]} 0 0
   [junit4]   2> 121108 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 121112 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[131]} 0 0
   [junit4]   2> 121115 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 121118 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[132]} 0 0
   [junit4]   2> 121121 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 121124 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[133]} 0 0
   [junit4]   2> 121127 INFO  (qtp1401126775-42) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[134]} 0 0
   [junit4]   2> 121130 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[134]} 0 0
   [junit4]   2> 121133 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[135]} 0 0
   [junit4]   2> 121136 INFO  (qtp865475013-62) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[135]} 0 0
   [junit4]   2> 121139 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 121142 INFO  (qtp865475013-66) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[136]} 0 0
   [junit4]   2> 121145 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 121148 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[137]} 0 0
   [junit4]   2> 121152 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 121163 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[138]} 0 0
   [junit4]   2> 121168 INFO  (qtp1401126775-45) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 121172 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[139]} 0 0
   [junit4]   2> 121181 INFO  (qtp1401126775-45) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 121184 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[140]} 0 0
   [junit4]   2> 121188 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 121191 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[141]} 0 0
   [junit4]   2> 121195 INFO  (qtp1401126775-49) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 121198 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[142]} 0 0
   [junit4]   2> 121211 INFO  (qtp1401126775-47) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[143]} 0 4
   [junit4]   2> 121220 INFO  (qtp865475013-64) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[143]} 0 4
   [junit4]   2> 121228 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 121238 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[144]} 0 0
   [junit4]   2> 121260 INFO  (qtp1401126775-48) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 121269 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[145]} 0 0
   [junit4]   2> 121280 INFO  (qtp1401126775-45) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 121284 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[146]} 0 0
   [junit4]   2> 121287 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 121290 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[147]} 0 0
   [junit4]   2> 121293 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 121296 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[148]} 0 0
   [junit4]   2> 121300 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 121303 INFO  (qtp865475013-59) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp= path=/update params={wt=javabin&version=2}{add=[149]} 0 0
   [junit4]   2> 121305 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 121637 INFO  (qtp1401126775-46) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=MockDirectoryWrapper(RAMDirectory@78588

[...truncated too long message...]

junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   32) Thread[id=64286, name=searcherExecutor-6025-thread-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   33) Thread[id=64181, name=searcherExecutor-5979-thread-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>   34) Thread[id=64255, name=commitScheduler-5983-thread-1, state=WAITING, group=TGRP-BasicDistributedZkTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
   [junit4]   2>         at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {regex_dup_A_s=PostingsFormat(name=LuceneFixedGap), other_tl1=PostingsFormat(name=LuceneVarGapFixedInterval), regex_dup_B_s=PostingsFormat(name=LuceneVarGapFixedInterval), range_facet_l_dv=Lucene50(blocksize=128), SubjectTerms_mfacet=PostingsFormat(name=LuceneVarGapFixedInterval), n_l1=PostingsFormat(name=LuceneVarGapFixedInterval), multiDefault=BlockTreeOrds(blocksize=128), intDefault=PostingsFormat(name=LuceneVarGapFixedInterval), genre_s=PostingsFormat(name=LuceneFixedGap), n_td1=BlockTreeOrds(blocksize=128), n_d1=PostingsFormat(name=LuceneVarGapFixedInterval), range_facet_l=PostingsFormat(name=LuceneVarGapFixedInterval), n_f1=BlockTreeOrds(blocksize=128), series_t=BlockTreeOrds(blocksize=128), n_tl1=BlockTreeOrds(blocksize=128), oddField_s=PostingsFormat(name=LuceneFixedGap), n_tf1=PostingsFormat(name=LuceneVarGapFixedInterval), price=PostingsFormat(name=LuceneFixedGap), a_t=PostingsFormat(name=LuceneVarGapFixedInterval), cat=PostingsFormat(name=LuceneVarGapFixedInterval), inStock=PostingsFormat(name=LuceneFixedGap), sequence_i=PostingsFormat(name=LuceneFixedGap), id=Lucene50(blocksize=128), text=PostingsFormat(name=LuceneFixedGap), timestamp=PostingsFormat(name=LuceneVarGapFixedInterval), foo_i=BlockTreeOrds(blocksize=128), val_i=Lucene50(blocksize=128), n_dt1=BlockTreeOrds(blocksize=128), author_t=BlockTreeOrds(blocksize=128), a_i1=PostingsFormat(name=LuceneVarGapFixedInterval), n_ti1=Lucene50(blocksize=128), rnd_b=Lucene50(blocksize=128), _version_=PostingsFormat(name=LuceneVarGapFixedInterval), foo_b=PostingsFormat(name=LuceneFixedGap), n_tdt1=PostingsFormat(name=LuceneVarGapFixedInterval), name=Lucene50(blocksize=128), foo_d=Lucene50(blocksize=128), range_facet_i_dv=PostingsFormat(name=LuceneVarGapFixedInterval), foo_f=PostingsFormat(name=LuceneFixedGap)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=505, maxMBSortInHeap=6.486188508070105, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=ms-MY, timezone=Africa/Bangui
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=55,free=335824096,total=529530880
   [junit4]   2> NOTE: All tests run in this JVM: [TestLMDirichletSimilarityFactory, DirectoryFactoryTest, TestMissingGroups, TestDistributedSearch, DisMaxRequestHandlerTest, CloneFieldUpdateProcessorFactoryTest, DistributedFacetPivotLongTailTest, JsonLoaderTest, HdfsLockFactoryTest, CloudExitableDirectoryReaderTest, TestFunctionQuery, URLClassifyProcessorTest, TestCloudInspectUtil, RAMDirectoryFactoryTest, TestExceedMaxTermLength, CursorPagingTest, AnalysisAfterCoreReloadTest, TestDefaultSearchFieldResource, DateFieldTest, TestMergePolicyConfig, ZkStateReaderTest, SignatureUpdateProcessorFactoryTest, TestShortCircuitedRequests, TestSolrQueryParserDefaultOperatorResource, TestTolerantUpdateProcessorCloud, TestDistribIDF, TestFaceting, TestRandomDVFaceting, CustomCollectionTest, TestHighlightDedupGrouping, QueryElevationComponentTest, ChangedSchemaMergeTest, TestImplicitCoreProperties, StandardRequestHandlerTest, ResponseLogComponentTest, TestIBSimilarityFactory, TestLRUCache, PreAnalyzedFieldTest, TestManagedResource, EnumFieldTest, IndexSchemaRuntimeFieldTest, TestCloudManagedSchema, TestAnalyzedSuggestions, DistributedQueueTest, TestFieldTypeResource, TestCrossCoreJoin, ExternalFileFieldSortTest, XsltUpdateRequestHandlerTest, CdcrReplicationDistributedZkTest, TestTestInjection, LeaderElectionIntegrationTest, TestSQLHandler, AddSchemaFieldsUpdateProcessorFactoryTest, SharedFSAutoReplicaFailoverUtilsTest, TestSha256AuthenticationProvider, CurrencyFieldOpenExchangeTest, DefaultValueUpdateProcessorTest, TestJsonFacets, TestFieldCollectionResource, RankQueryTest, TriLevelCompositeIdRoutingTest, DeleteReplicaTest, CacheHeaderTest, ConnectionReuseTest, TestMiniSolrCloudClusterSSL, DistributedFacetPivotWhiteBoxTest, TestOmitPositions, JSONWriterTest, ConcurrentDeleteAndCreateCollectionTest, TestCollationFieldDocValues, TestComponentsName, QueryResultKeyTest, DistributedQueryElevationComponentTest, TestDocSet, TestCustomSort, TestIntervalFaceting, TestCSVLoader, DistribCursorPagingTest, TolerantUpdateProcessorTest, EchoParamsTest, TestArbitraryIndexDir, TestFreeTextSuggestions, TestSearcherReuse, TestSolr4Spatial2, DistributedSpellCheckComponentTest, TestReplicationHandlerBackup, ZkNodePropsTest, TestReRankQParserPlugin, HttpPartitionTest, TestConfigSetProperties, TestObjectReleaseTracker, OutputWriterTest, PingRequestHandlerTest, AnalyticsQueryTest, TestRandomFaceting, CollectionTooManyReplicasTest, CdcrVersionReplicationTest, BasicDistributedZkTest]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=BasicDistributedZkTest -Dtests.seed=26A2A1B7F3AAB02 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=ms-MY -Dtests.timezone=Africa/Bangui -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | BasicDistributedZkTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([26A2A1B7F3AAB02]:0)
   [junit4] Completed [585/585 (4!)] on J0 in 7225.13s, 1 test, 2 errors <<< FAILURES!

[...truncated 15 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2363 tests, 1 suite-level error, 2 errors, 2 failures, 31 ignored (16 assumptions) [seed: 26A2A1B7F3AAB02]

Total time: 381 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 35 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/35/

5 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:43556

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:43556
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:439)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:
Timeout waiting for all live and active

Stack Trace:
java.lang.AssertionError: Timeout waiting for all live and active
	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C:4F471EFA74C4F194]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:193)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=82600, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=82600, name=collection1, state=RUNNABLE, group=TGRP-HdfsCollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:57045: collection already exists: awholynewstresscollection_collection1_0
	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1593)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1614)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload

Error Message:
expected:<[{indexVersion=1460370822447,generation=2,filelist=[_3g.fdt, _3g.fdx, _3g.fnm, _3g.nvd, _3g.nvm, _3g.si, _3g_Asserting_0.doc, _3g_Asserting_0.tim, _3g_Asserting_0.tip, _62.fdt, _62.fdx, _62.fnm, _62.nvd, _62.nvm, _62.si, _62_Asserting_0.doc, _62_Asserting_0.tim, _62_Asserting_0.tip, _63.cfe, _63.cfs, _63.si, _64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_Asserting_0.doc, _64_Asserting_0.tim, _64_Asserting_0.tip, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_Asserting_0.doc, _65_Asserting_0.tim, _65_Asserting_0.tip, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_Asserting_0.doc, _66_Asserting_0.tim, _66_Asserting_0.tip, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_Asserting_0.doc, _67_Asserting_0.tim, _67_Asserting_0.tip, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_Asserting_0.doc, _68_Asserting_0.tim, _68_Asserting_0.tip, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_Asserting_0.doc, _69_Asserting_0.tim, _69_Asserting_0.tip, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_Asserting_0.doc, _6a_Asserting_0.tim, _6a_Asserting_0.tip, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_Asserting_0.doc, _6b_Asserting_0.tim, _6b_Asserting_0.tip, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_Asserting_0.doc, _6c_Asserting_0.tim, _6c_Asserting_0.tip, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_Asserting_0.doc, _6d_Asserting_0.tim, _6d_Asserting_0.tip, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_Asserting_0.doc, _6e_Asserting_0.tim, _6e_Asserting_0.tip, _6f.fdt, _6f.fdx, _6f.fnm, _6f.nvd, _6f.nvm, _6f.si, _6f_Asserting_0.doc, _6f_Asserting_0.tim, _6f_Asserting_0.tip, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_Asserting_0.doc, _6g_Asserting_0.tim, _6g_Asserting_0.tip, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_Asserting_0.doc, _6h_Asserting_0.tim, _6h_Asserting_0.tip, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_Asserting_0.doc, _6i_Asserting_0.tim, _6i_Asserting_0.tip, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_Asserting_0.doc, _6j_Asserting_0.tim, _6j_Asserting_0.tip, _6k.fdt, _6k.fdx, _6k.fnm, _6k.nvd, _6k.nvm, _6k.si, _6k_Asserting_0.doc, _6k_Asserting_0.tim, _6k_Asserting_0.tip, _6l.fdt, _6l.fdx, _6l.fnm, _6l.nvd, _6l.nvm, _6l.si, _6l_Asserting_0.doc, _6l_Asserting_0.tim, _6l_Asserting_0.tip, _6m.fdt, _6m.fdx, _6m.fnm, _6m.nvd, _6m.nvm, _6m.si, _6m_Asserting_0.doc, _6m_Asserting_0.tim, _6m_Asserting_0.tip, _6n.fdt, _6n.fdx, _6n.fnm, _6n.nvd, _6n.nvm, _6n.si, _6n_Asserting_0.doc, _6n_Asserting_0.tim, _6n_Asserting_0.tip, _6o.fdt, _6o.fdx, _6o.fnm, _6o.nvd, _6o.nvm, _6o.si, _6o_Asserting_0.doc, _6o_Asserting_0.tim, _6o_Asserting_0.tip, _6p.fdt, _6p.fdx, _6p.fnm, _6p.nvd, _6p.nvm, _6p.si, _6p_Asserting_0.doc, _6p_Asserting_0.tim, _6p_Asserting_0.tip, _6q.fdt, _6q.fdx, _6q.fnm, _6q.nvd, _6q.nvm, _6q.si, _6q_Asserting_0.doc, _6q_Asserting_0.tim, _6q_Asserting_0.tip, _6r.fdt, _6r.fdx, _6r.fnm, _6r.nvd, _6r.nvm, _6r.si, _6r_Asserting_0.doc, _6r_Asserting_0.tim, _6r_Asserting_0.tip, _6s.fdt, _6s.fdx, _6s.fnm, _6s.nvd, _6s.nvm, _6s.si, _6s_Asserting_0.doc, _6s_Asserting_0.tim, _6s_Asserting_0.tip, _6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Asserting_0.doc, _6t_Asserting_0.tim, _6t_Asserting_0.tip, _6u.fdt, _6u.fdx, _6u.fnm, _6u.nvd, _6u.nvm, _6u.si, _6u_Asserting_0.doc, _6u_Asserting_0.tim, _6u_Asserting_0.tip, _6v.fdt, _6v.fdx, _6v.fnm, _6v.nvd, _6v.nvm, _6v.si, _6v_Asserting_0.doc, _6v_Asserting_0.tim, _6v_Asserting_0.tip, _6w.fdt, _6w.fdx, _6w.fnm, _6w.nvd, _6w.nvm, _6w.si, _6w_Asserting_0.doc, _6w_Asserting_0.tim, _6w_Asserting_0.tip, _6x.fdt, _6x.fdx, _6x.fnm, _6x.nvd, _6x.nvm, _6x.si, _6x_Asserting_0.doc, _6x_Asserting_0.tim, _6x_Asserting_0.tip, segments_2]}]> but was:<[{indexVersion=1460370822447,generation=2,filelist=[_3g.fdt, _3g.fdx, _3g.fnm, _3g.nvd, _3g.nvm, _3g.si, _3g_Asserting_0.doc, _3g_Asserting_0.tim, _3g_Asserting_0.tip, _62.fdt, _62.fdx, _62.fnm, _62.nvd, _62.nvm, _62.si, _62_Asserting_0.doc, _62_Asserting_0.tim, _62_Asserting_0.tip, _63.cfe, _63.cfs, _63.si, _64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_Asserting_0.doc, _64_Asserting_0.tim, _64_Asserting_0.tip, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_Asserting_0.doc, _65_Asserting_0.tim, _65_Asserting_0.tip, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_Asserting_0.doc, _66_Asserting_0.tim, _66_Asserting_0.tip, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_Asserting_0.doc, _67_Asserting_0.tim, _67_Asserting_0.tip, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_Asserting_0.doc, _68_Asserting_0.tim, _68_Asserting_0.tip, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_Asserting_0.doc, _69_Asserting_0.tim, _69_Asserting_0.tip, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_Asserting_0.doc, _6a_Asserting_0.tim, _6a_Asserting_0.tip, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_Asserting_0.doc, _6b_Asserting_0.tim, _6b_Asserting_0.tip, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_Asserting_0.doc, _6c_Asserting_0.tim, _6c_Asserting_0.tip, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_Asserting_0.doc, _6d_Asserting_0.tim, _6d_Asserting_0.tip, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_Asserting_0.doc, _6e_Asserting_0.tim, _6e_Asserting_0.tip, _6f.fdt, _6f.fdx, _6f.fnm, _6f.nvd, _6f.nvm, _6f.si, _6f_Asserting_0.doc, _6f_Asserting_0.tim, _6f_Asserting_0.tip, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_Asserting_0.doc, _6g_Asserting_0.tim, _6g_Asserting_0.tip, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_Asserting_0.doc, _6h_Asserting_0.tim, _6h_Asserting_0.tip, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_Asserting_0.doc, _6i_Asserting_0.tim, _6i_Asserting_0.tip, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_Asserting_0.doc, _6j_Asserting_0.tim, _6j_Asserting_0.tip, _6k.fdt, _6k.fdx, _6k.fnm, _6k.nvd, _6k.nvm, _6k.si, _6k_Asserting_0.doc, _6k_Asserting_0.tim, _6k_Asserting_0.tip, _6l.fdt, _6l.fdx, _6l.fnm, _6l.nvd, _6l.nvm, _6l.si, _6l_Asserting_0.doc, _6l_Asserting_0.tim, _6l_Asserting_0.tip, _6m.fdt, _6m.fdx, _6m.fnm, _6m.nvd, _6m.nvm, _6m.si, _6m_Asserting_0.doc, _6m_Asserting_0.tim, _6m_Asserting_0.tip, _6n.fdt, _6n.fdx, _6n.fnm, _6n.nvd, _6n.nvm, _6n.si, _6n_Asserting_0.doc, _6n_Asserting_0.tim, _6n_Asserting_0.tip, _6o.fdt, _6o.fdx, _6o.fnm, _6o.nvd, _6o.nvm, _6o.si, _6o_Asserting_0.doc, _6o_Asserting_0.tim, _6o_Asserting_0.tip, _6p.fdt, _6p.fdx, _6p.fnm, _6p.nvd, _6p.nvm, _6p.si, _6p_Asserting_0.doc, _6p_Asserting_0.tim, _6p_Asserting_0.tip, _6q.fdt, _6q.fdx, _6q.fnm, _6q.nvd, _6q.nvm, _6q.si, _6q_Asserting_0.doc, _6q_Asserting_0.tim, _6q_Asserting_0.tip, _6r.fdt, _6r.fdx, _6r.fnm, _6r.nvd, _6r.nvm, _6r.si, _6r_Asserting_0.doc, _6r_Asserting_0.tim, _6r_Asserting_0.tip, _6s.fdt, _6s.fdx, _6s.fnm, _6s.nvd, _6s.nvm, _6s.si, _6s_Asserting_0.doc, _6s_Asserting_0.tim, _6s_Asserting_0.tip, _6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Asserting_0.doc, _6t_Asserting_0.tim, _6t_Asserting_0.tip, _6u.fdt, _6u.fdx, _6u.fnm, _6u.nvd, _6u.nvm, _6u.si, _6u_Asserting_0.doc, _6u_Asserting_0.tim, _6u_Asserting_0.tip, _6v.fdt, _6v.fdx, _6v.fnm, _6v.nvd, _6v.nvm, _6v.si, _6v_Asserting_0.doc, _6v_Asserting_0.tim, _6v_Asserting_0.tip, _6w.fdt, _6w.fdx, _6w.fnm, _6w.nvd, _6w.nvm, _6w.si, _6w_Asserting_0.doc, _6w_Asserting_0.tim, _6w_Asserting_0.tip, _6x.fdt, _6x.fdx, _6x.fnm, _6x.nvd, _6x.nvm, _6x.si, _6x_Asserting_0.doc, _6x_Asserting_0.tim, _6x_Asserting_0.tip, segments_2]}, {indexVersion=1460370822447,generation=3,filelist=[_3g.fdt, _3g.fdx, _3g.fnm, _3g.nvd, _3g.nvm, _3g.si, _3g_Asserting_0.doc, _3g_Asserting_0.tim, _3g_Asserting_0.tip, _6x.fdt, _6x.fdx, _6x.fnm, _6x.nvd, _6x.nvm, _6x.si, _6x_Asserting_0.doc, _6x_Asserting_0.tim, _6x_Asserting_0.tip, _6y.cfe, _6y.cfs, _6y.si, segments_3]}]>

Stack Trace:
java.lang.AssertionError: expected:<[{indexVersion=1460370822447,generation=2,filelist=[_3g.fdt, _3g.fdx, _3g.fnm, _3g.nvd, _3g.nvm, _3g.si, _3g_Asserting_0.doc, _3g_Asserting_0.tim, _3g_Asserting_0.tip, _62.fdt, _62.fdx, _62.fnm, _62.nvd, _62.nvm, _62.si, _62_Asserting_0.doc, _62_Asserting_0.tim, _62_Asserting_0.tip, _63.cfe, _63.cfs, _63.si, _64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_Asserting_0.doc, _64_Asserting_0.tim, _64_Asserting_0.tip, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_Asserting_0.doc, _65_Asserting_0.tim, _65_Asserting_0.tip, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_Asserting_0.doc, _66_Asserting_0.tim, _66_Asserting_0.tip, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_Asserting_0.doc, _67_Asserting_0.tim, _67_Asserting_0.tip, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_Asserting_0.doc, _68_Asserting_0.tim, _68_Asserting_0.tip, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_Asserting_0.doc, _69_Asserting_0.tim, _69_Asserting_0.tip, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_Asserting_0.doc, _6a_Asserting_0.tim, _6a_Asserting_0.tip, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_Asserting_0.doc, _6b_Asserting_0.tim, _6b_Asserting_0.tip, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_Asserting_0.doc, _6c_Asserting_0.tim, _6c_Asserting_0.tip, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_Asserting_0.doc, _6d_Asserting_0.tim, _6d_Asserting_0.tip, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_Asserting_0.doc, _6e_Asserting_0.tim, _6e_Asserting_0.tip, _6f.fdt, _6f.fdx, _6f.fnm, _6f.nvd, _6f.nvm, _6f.si, _6f_Asserting_0.doc, _6f_Asserting_0.tim, _6f_Asserting_0.tip, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_Asserting_0.doc, _6g_Asserting_0.tim, _6g_Asserting_0.tip, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_Asserting_0.doc, _6h_Asserting_0.tim, _6h_Asserting_0.tip, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_Asserting_0.doc, _6i_Asserting_0.tim, _6i_Asserting_0.tip, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_Asserting_0.doc, _6j_Asserting_0.tim, _6j_Asserting_0.tip, _6k.fdt, _6k.fdx, _6k.fnm, _6k.nvd, _6k.nvm, _6k.si, _6k_Asserting_0.doc, _6k_Asserting_0.tim, _6k_Asserting_0.tip, _6l.fdt, _6l.fdx, _6l.fnm, _6l.nvd, _6l.nvm, _6l.si, _6l_Asserting_0.doc, _6l_Asserting_0.tim, _6l_Asserting_0.tip, _6m.fdt, _6m.fdx, _6m.fnm, _6m.nvd, _6m.nvm, _6m.si, _6m_Asserting_0.doc, _6m_Asserting_0.tim, _6m_Asserting_0.tip, _6n.fdt, _6n.fdx, _6n.fnm, _6n.nvd, _6n.nvm, _6n.si, _6n_Asserting_0.doc, _6n_Asserting_0.tim, _6n_Asserting_0.tip, _6o.fdt, _6o.fdx, _6o.fnm, _6o.nvd, _6o.nvm, _6o.si, _6o_Asserting_0.doc, _6o_Asserting_0.tim, _6o_Asserting_0.tip, _6p.fdt, _6p.fdx, _6p.fnm, _6p.nvd, _6p.nvm, _6p.si, _6p_Asserting_0.doc, _6p_Asserting_0.tim, _6p_Asserting_0.tip, _6q.fdt, _6q.fdx, _6q.fnm, _6q.nvd, _6q.nvm, _6q.si, _6q_Asserting_0.doc, _6q_Asserting_0.tim, _6q_Asserting_0.tip, _6r.fdt, _6r.fdx, _6r.fnm, _6r.nvd, _6r.nvm, _6r.si, _6r_Asserting_0.doc, _6r_Asserting_0.tim, _6r_Asserting_0.tip, _6s.fdt, _6s.fdx, _6s.fnm, _6s.nvd, _6s.nvm, _6s.si, _6s_Asserting_0.doc, _6s_Asserting_0.tim, _6s_Asserting_0.tip, _6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Asserting_0.doc, _6t_Asserting_0.tim, _6t_Asserting_0.tip, _6u.fdt, _6u.fdx, _6u.fnm, _6u.nvd, _6u.nvm, _6u.si, _6u_Asserting_0.doc, _6u_Asserting_0.tim, _6u_Asserting_0.tip, _6v.fdt, _6v.fdx, _6v.fnm, _6v.nvd, _6v.nvm, _6v.si, _6v_Asserting_0.doc, _6v_Asserting_0.tim, _6v_Asserting_0.tip, _6w.fdt, _6w.fdx, _6w.fnm, _6w.nvd, _6w.nvm, _6w.si, _6w_Asserting_0.doc, _6w_Asserting_0.tim, _6w_Asserting_0.tip, _6x.fdt, _6x.fdx, _6x.fnm, _6x.nvd, _6x.nvm, _6x.si, _6x_Asserting_0.doc, _6x_Asserting_0.tim, _6x_Asserting_0.tip, segments_2]}]> but was:<[{indexVersion=1460370822447,generation=2,filelist=[_3g.fdt, _3g.fdx, _3g.fnm, _3g.nvd, _3g.nvm, _3g.si, _3g_Asserting_0.doc, _3g_Asserting_0.tim, _3g_Asserting_0.tip, _62.fdt, _62.fdx, _62.fnm, _62.nvd, _62.nvm, _62.si, _62_Asserting_0.doc, _62_Asserting_0.tim, _62_Asserting_0.tip, _63.cfe, _63.cfs, _63.si, _64.fdt, _64.fdx, _64.fnm, _64.nvd, _64.nvm, _64.si, _64_Asserting_0.doc, _64_Asserting_0.tim, _64_Asserting_0.tip, _65.fdt, _65.fdx, _65.fnm, _65.nvd, _65.nvm, _65.si, _65_Asserting_0.doc, _65_Asserting_0.tim, _65_Asserting_0.tip, _66.fdt, _66.fdx, _66.fnm, _66.nvd, _66.nvm, _66.si, _66_Asserting_0.doc, _66_Asserting_0.tim, _66_Asserting_0.tip, _67.fdt, _67.fdx, _67.fnm, _67.nvd, _67.nvm, _67.si, _67_Asserting_0.doc, _67_Asserting_0.tim, _67_Asserting_0.tip, _68.fdt, _68.fdx, _68.fnm, _68.nvd, _68.nvm, _68.si, _68_Asserting_0.doc, _68_Asserting_0.tim, _68_Asserting_0.tip, _69.fdt, _69.fdx, _69.fnm, _69.nvd, _69.nvm, _69.si, _69_Asserting_0.doc, _69_Asserting_0.tim, _69_Asserting_0.tip, _6a.fdt, _6a.fdx, _6a.fnm, _6a.nvd, _6a.nvm, _6a.si, _6a_Asserting_0.doc, _6a_Asserting_0.tim, _6a_Asserting_0.tip, _6b.fdt, _6b.fdx, _6b.fnm, _6b.nvd, _6b.nvm, _6b.si, _6b_Asserting_0.doc, _6b_Asserting_0.tim, _6b_Asserting_0.tip, _6c.fdt, _6c.fdx, _6c.fnm, _6c.nvd, _6c.nvm, _6c.si, _6c_Asserting_0.doc, _6c_Asserting_0.tim, _6c_Asserting_0.tip, _6d.fdt, _6d.fdx, _6d.fnm, _6d.nvd, _6d.nvm, _6d.si, _6d_Asserting_0.doc, _6d_Asserting_0.tim, _6d_Asserting_0.tip, _6e.fdt, _6e.fdx, _6e.fnm, _6e.nvd, _6e.nvm, _6e.si, _6e_Asserting_0.doc, _6e_Asserting_0.tim, _6e_Asserting_0.tip, _6f.fdt, _6f.fdx, _6f.fnm, _6f.nvd, _6f.nvm, _6f.si, _6f_Asserting_0.doc, _6f_Asserting_0.tim, _6f_Asserting_0.tip, _6g.fdt, _6g.fdx, _6g.fnm, _6g.nvd, _6g.nvm, _6g.si, _6g_Asserting_0.doc, _6g_Asserting_0.tim, _6g_Asserting_0.tip, _6h.fdt, _6h.fdx, _6h.fnm, _6h.nvd, _6h.nvm, _6h.si, _6h_Asserting_0.doc, _6h_Asserting_0.tim, _6h_Asserting_0.tip, _6i.fdt, _6i.fdx, _6i.fnm, _6i.nvd, _6i.nvm, _6i.si, _6i_Asserting_0.doc, _6i_Asserting_0.tim, _6i_Asserting_0.tip, _6j.fdt, _6j.fdx, _6j.fnm, _6j.nvd, _6j.nvm, _6j.si, _6j_Asserting_0.doc, _6j_Asserting_0.tim, _6j_Asserting_0.tip, _6k.fdt, _6k.fdx, _6k.fnm, _6k.nvd, _6k.nvm, _6k.si, _6k_Asserting_0.doc, _6k_Asserting_0.tim, _6k_Asserting_0.tip, _6l.fdt, _6l.fdx, _6l.fnm, _6l.nvd, _6l.nvm, _6l.si, _6l_Asserting_0.doc, _6l_Asserting_0.tim, _6l_Asserting_0.tip, _6m.fdt, _6m.fdx, _6m.fnm, _6m.nvd, _6m.nvm, _6m.si, _6m_Asserting_0.doc, _6m_Asserting_0.tim, _6m_Asserting_0.tip, _6n.fdt, _6n.fdx, _6n.fnm, _6n.nvd, _6n.nvm, _6n.si, _6n_Asserting_0.doc, _6n_Asserting_0.tim, _6n_Asserting_0.tip, _6o.fdt, _6o.fdx, _6o.fnm, _6o.nvd, _6o.nvm, _6o.si, _6o_Asserting_0.doc, _6o_Asserting_0.tim, _6o_Asserting_0.tip, _6p.fdt, _6p.fdx, _6p.fnm, _6p.nvd, _6p.nvm, _6p.si, _6p_Asserting_0.doc, _6p_Asserting_0.tim, _6p_Asserting_0.tip, _6q.fdt, _6q.fdx, _6q.fnm, _6q.nvd, _6q.nvm, _6q.si, _6q_Asserting_0.doc, _6q_Asserting_0.tim, _6q_Asserting_0.tip, _6r.fdt, _6r.fdx, _6r.fnm, _6r.nvd, _6r.nvm, _6r.si, _6r_Asserting_0.doc, _6r_Asserting_0.tim, _6r_Asserting_0.tip, _6s.fdt, _6s.fdx, _6s.fnm, _6s.nvd, _6s.nvm, _6s.si, _6s_Asserting_0.doc, _6s_Asserting_0.tim, _6s_Asserting_0.tip, _6t.fdt, _6t.fdx, _6t.fnm, _6t.nvd, _6t.nvm, _6t.si, _6t_Asserting_0.doc, _6t_Asserting_0.tim, _6t_Asserting_0.tip, _6u.fdt, _6u.fdx, _6u.fnm, _6u.nvd, _6u.nvm, _6u.si, _6u_Asserting_0.doc, _6u_Asserting_0.tim, _6u_Asserting_0.tip, _6v.fdt, _6v.fdx, _6v.fnm, _6v.nvd, _6v.nvm, _6v.si, _6v_Asserting_0.doc, _6v_Asserting_0.tim, _6v_Asserting_0.tip, _6w.fdt, _6w.fdx, _6w.fnm, _6w.nvd, _6w.nvm, _6w.si, _6w_Asserting_0.doc, _6w_Asserting_0.tim, _6w_Asserting_0.tip, _6x.fdt, _6x.fdx, _6x.fnm, _6x.nvd, _6x.nvm, _6x.si, _6x_Asserting_0.doc, _6x_Asserting_0.tim, _6x_Asserting_0.tip, segments_2]}, {indexVersion=1460370822447,generation=3,filelist=[_3g.fdt, _3g.fdx, _3g.fnm, _3g.nvd, _3g.nvm, _3g.si, _3g_Asserting_0.doc, _3g_Asserting_0.tim, _3g_Asserting_0.tip, _6x.fdt, _6x.fdx, _6x.fnm, _6x.nvd, _6x.nvm, _6x.si, _6x_Asserting_0.doc, _6x_Asserting_0.tim, _6x_Asserting_0.tip, _6y.cfe, _6y.cfs, _6y.si, segments_3]}]>
	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C:E2C43A10AA70926F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:147)
	at org.apache.solr.handler.TestReplicationHandler.doTestReplicateAfterCoreReload(TestReplicationHandler.java:1144)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication

Error Message:
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-011/./collection1/data/index.20160411223406535, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-011/./collection1/data/index, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-011/./collection1/data] expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-011/./collection1/data/index.20160411223406535, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-011/./collection1/data/index, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-011/./collection1/data] expected:<2> but was:<3>
	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C:D35B7A75F93F2172]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.handler.TestReplicationHandler.checkForSingleIndex(TestReplicationHandler.java:819)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigReplication(TestReplicationHandler.java:534)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11520 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReplicationHandler
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/init-core-data-001
   [junit4]   2> 209517 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testEmptyCommits
   [junit4]   2> 209517 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1
   [junit4]   2> 209526 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 209541 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@41aa53a0{/solr,null,AVAILABLE}
   [junit4]   2> 209546 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@6c541e99{HTTP/1.1,[http/1.1]}{127.0.0.1:34681}
   [junit4]   2> 209546 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.Server Started @215985ms
   [junit4]   2> 209546 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1/data, hostContext=/solr, hostPort=34681}
   [junit4]   2> 209547 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 209547 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001'
   [junit4]   2> 209547 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 209547 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 209547 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/solr.xml
   [junit4]   2> 209550 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/.
   [junit4]   2> 209550 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer New CoreContainer 1130827808
   [junit4]   2> 209550 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001]
   [junit4]   2> 209550 WARN  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/lib
   [junit4]   2> 209558 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 209559 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 209559 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 209559 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 209559 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 209559 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 209560 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/.
   [junit4]   2> 209561 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 209567 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1
   [junit4]   2> 209567 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 209568 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1'
   [junit4]   2> 209568 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 209568 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 209577 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 209577 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 209578 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 209580 WARN  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 209581 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 209589 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 209591 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 209601 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 209601 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1'
   [junit4]   2> 209601 INFO  (coreLoadExecutor-549-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1
   [junit4]   2> 209602 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 209602 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1/data/]
   [junit4]   2> 209602 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 209602 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1/data
   [junit4]   2> 209602 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1/data/index/
   [junit4]   2> 209602 WARN  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 209603 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1/data/index
   [junit4]   2> 209603 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.31640625, noCFSRatio=0.0]
   [junit4]   2> 209620 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 209621 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 209622 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 209623 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 209623 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 209624 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 209624 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 209629 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 209629 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/admin/segments,/admin/system,/replication,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file
   [junit4]   2> 209629 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 209631 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/./collection1/data/
   [junit4]   2> 209633 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 209633 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 209633 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.840142715649478]
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@73b617eb[collection1] main]
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1/conf
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1/conf}
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-001/collection1/conf
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Replication enabled for following config files: schema.xml,xslt/dummy.xsl
   [junit4]   2> 209634 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 209635 INFO  (coreLoadExecutor-549-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 209637 INFO  (searcherExecutor-550-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@73b617eb[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 210081 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1
   [junit4]   2> 210083 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@548d2022{/solr,null,AVAILABLE}
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@3993e0e8{HTTP/1.1,[http/1.1]}{127.0.0.1:40623}
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.e.j.s.Server Started @216548ms
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1/data, hostContext=/solr, hostPort=40623}
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002'
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 210109 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 210110 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/solr.xml
   [junit4]   2> 210172 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/.
   [junit4]   2> 210173 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer New CoreContainer 1176521909
   [junit4]   2> 210173 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002]
   [junit4]   2> 210173 WARN  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/lib
   [junit4]   2> 210174 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 210174 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 210175 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 210175 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 210175 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 210175 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 210175 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/.
   [junit4]   2> 210176 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, loadOnStartup=true, schema=schema.xml, configSetProperties=configsetprops.json, transient=false, dataDir=data/}
   [junit4]   2> 210176 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1
   [junit4]   2> 210176 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 210182 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1'
   [junit4]   2> 210182 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 210183 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 210184 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 210185 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 210197 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 210199 WARN  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 210199 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 210204 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 210206 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 210223 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1'
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    ] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1], dataDir=[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/]
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore JMX monitoring not detected for core: collection1
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data
   [junit4]   2> 210225 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index/
   [junit4]   2> 210225 WARN  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 210226 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index
   [junit4]   2> 210226 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=47, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=0.31640625, noCFSRatio=0.0]
   [junit4]   2> 210232 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 210232 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 210249 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [    x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/
   [junit4]   2> 210250 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 210254 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 210255 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 210320 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 210321 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 210325 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.PluginBag requestHandler : 'lazy' created with startup=lazy 
   [junit4]   2> 210325 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,lazy,/update/json/docs,/admin/luke,/export,/get,/admin/properties,/update/json,/admin/threads,/update/csv,/sql,/admin/segments,/admin/system,/replication,defaults,/config,/stream,/schema,/admin/plugins,/admin/logging,/admin/ping,/update,/admin/file
   [junit4]   2> 210325 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 210327 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 210327 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 210328 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=31, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.840142715649478]
   [junit4]   2> 210328 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 210328 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 210329 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@7cee5843[collection1] main]
   [junit4]   2> 210329 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1/conf
   [junit4]   2> 210329 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {storageDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1/conf}
   [junit4]   2> 210329 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using file:dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/collection1/conf
   [junit4]   2> 210329 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 210331 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 1000ms
   [junit4]   2> 210331 INFO  (coreLoadExecutor-559-thread-1) [    x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 210331 INFO  (searcherExecutor-560-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@7cee5843[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 210730 INFO  (qtp1855534809-1017) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=*:*&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 210757 INFO  (qtp1855534809-1017) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={wt=javabin&version=2}{add=[1]} 0 24
   [junit4]   2> 210765 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 210924 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=2
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-001 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_2,generation=2}
   [junit4]   2> 210941 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 210959 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@5d6da35a[collection1] main]
   [junit4]   2> 210959 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.u.DirectUpdateHandler2 end_commit_flush
   [junit4]   2> 210969 INFO  (searcherExecutor-550-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@5d6da35a[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.1.0):C1)))}
   [junit4]   2> 210973 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.u.p.LogUpdateProcessorFactory [collection1]  webapp=/solr path=/update params={commit=true&wt=javabin&version=2}{commit=} 0 208
   [junit4]   2> 210985 INFO  (qtp1855534809-1015) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 210985 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Master's generation: 2
   [junit4]   2> 210985 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Slave's generation: 1
   [junit4]   2> 210985 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Starting replication process
   [junit4]   2> 211007 INFO  (qtp1855534809-1014) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&wt=javabin&version=2&command=filelist} status=0 QTime=6
   [junit4]   2> 211008 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Number of files in latest index in master: 10
   [junit4]   2> 211016 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index.20160411222153777
   [junit4]   2> 211017 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 211047 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@6b2667fd
   [junit4]   2> 211048 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 211056 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 211056 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 211056 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 211056 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 211059 INFO  (qtp749871808-1034) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={wait=true&masterUrl=http://127.0.0.1:34681/solr/collection1/replication&command=fetchindex} status=0 QTime=1
   [junit4]   2> 211059 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter... core=collection1
   [junit4]   2> 211060 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Starting download (fullCopy=false) to MockDirectoryWrapper(RAMDirectory@53234e3f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3a1983e2)
   [junit4]   2> 211062 INFO  (qtp1855534809-1020) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.si&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211065 INFO  (qtp1855534809-1017) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvd&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211068 INFO  (qtp1855534809-1016) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdx&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211071 INFO  (qtp749871808-1032) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=name:empty1&sort=id+desc&wt=javabin&version=2} hits=0 status=0 QTime=0
   [junit4]   2> 211072 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.h.TestReplicationHandler Waiting for 1 docs
   [junit4]   2> 211072 INFO  (qtp1855534809-1015) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.doc&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211075 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.tim&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211078 INFO  (qtp1855534809-1014) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fdt&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211080 INFO  (qtp1855534809-1020) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0_Asserting_0.tip&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211082 INFO  (qtp1855534809-1020) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.fnm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211085 INFO  (qtp1855534809-1016) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=_0.nvm&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211088 INFO  (qtp1855534809-1015) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={generation=2&qt=/replication&file=segments_2&checksum=true&wt=filestream&command=filecontent} status=0 QTime=0
   [junit4]   2> 211088 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.h.IndexFetcher Total time taken for download (fullCopy=false,bytesDownloaded=1629) : 0 secs (null bytes/sec) to MockDirectoryWrapper(RAMDirectory@53234e3f lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@3a1983e2)
   [junit4]   2> 211092 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=23, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 211093 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=2
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_1,generation=1}
   [junit4]   2> 	commit{dir=RawDirectoryWrapper(NIOFSDirectory@/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/index-NIOFSDirectory-002 lockFactory=org.apache.lucene.store.NativeFSLockFactory@497a095e),segFN=segments_2,generation=2}
   [junit4]   2> 211093 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 2
   [junit4]   2> 211094 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 211095 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@3e25be90[collection1] main]
   [junit4]   2> 211096 INFO  (searcherExecutor-560-thread-1-processing-x:collection1) [    x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@3e25be90[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_0(6.1.0):C1)))}
   [junit4]   2> 211096 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory looking to close /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index.20160411222153777 [CachedDir<<refCount=0;path=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index.20160411222153777;done=true>>]
   [junit4]   2> 211096 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory, CoreContainer#isShutdown=false
   [junit4]   2> 211096 INFO  (indexFetcher-564-thread-1) [    x:collection1] o.a.s.c.CachingDirectoryFactory Closing directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.handler.TestReplicationHandler_C7132120DA389C6C-001/solr-instance-002/./collection1/data/index.20160411222153777
   [junit4]   2> 211185 INFO  (qtp749871808-1032) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/select params={q=name:empty1&sort=id+desc&wt=javabin&version=2} hits=1 status=0 QTime=7
   [junit4]   2> 211193 INFO  (TEST-TestReplicationHandler.testEmptyCommits-seed#[C7132120DA389C6C]) [    ] o.a.s.h.TestReplicationHandler Waited for 100ms and found 1 docs
   [junit4]   2> 211198 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 211205 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 211217 INFO  (qtp1855534809-1019) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={slave=false&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
   [junit4]   2> 211218 INFO  (qtp749871808-1038) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=getDetails&qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=8
   [junit4]   2> 211221 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 211233 INFO  (qtp749871808-1031) [    x:collection1] o.a.s.c.S.Request [collection1]  webapp=/solr path=/replication params={_trace=assertVersions&qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
   [junit4]   2> 211237 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.u.DirectUpdateHandler2 start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
   [junit4]   2> 211237 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
   [junit4]   2> 211237 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.c.SolrCore SolrIndexSearcher has not changed - not re-opening: org.apache.solr.search.SolrIndexSearcher
   [junit4]   2> 211238 INFO  (qtp1855534809-1021) [    x:collection1] o.a.s.u.Dire

[...truncated too long message...]

e.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
   [junit4]    > 	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
   [junit4]    > 	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
   [junit4]    > 	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
   [junit4]    > 	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
   [junit4]    > 	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
   [junit4]    > 	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
   [junit4]    > 	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
   [junit4]    > 	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
   [junit4]    > 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
   [junit4]    > 	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
   [junit4]    > 	... 11 moreThrowable #8: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25363, name=NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C]:0)
   [junit4]    > 	at java.util.AbstractCollection.toArray(AbstractCollection.java:136)
   [junit4]    > 	at sun.nio.ch.Util$2.toArray(Util.java:240)
   [junit4]    > 	at java.util.ArrayList.<init>(ArrayList.java:177)
   [junit4]    > 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:183)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #9: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=25364, name=SessionTracker, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C]:0)
   [junit4]    > 	at java.lang.Long.valueOf(Long.java:840)
   [junit4]    > 	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:150)Throwable #10: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=29416, name=zkCallback-1551-thread-15, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C7132120DA389C6C]:0)
   [junit4]    > 	at java.lang.ThreadLocal$ThreadLocalMap.set(ThreadLocal.java:481)
   [junit4]    > 	at java.lang.ThreadLocal$ThreadLocalMap.access$100(ThreadLocal.java:298)
   [junit4]    > 	at java.lang.ThreadLocal.set(ThreadLocal.java:203)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:215)
   [junit4]    > 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$7/200831499.run(Unknown Source)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   [junit4]    > 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 7026713 INFO  (SUITE-CollectionsAPIDistributedZkTest-seed#[C7132120DA389C6C]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.CollectionsAPIDistributedZkTest_C7132120DA389C6C-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=PostingsFormat(name=Direct), range_facet_l_dv=Lucene50(blocksize=128), _version_=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=Asserting), a_t=PostingsFormat(name=Direct), intDefault=PostingsFormat(name=Direct), a_i1=PostingsFormat(name=Direct), id=Lucene50(blocksize=128), range_facet_i_dv=PostingsFormat(name=Direct), text=FST50, range_facet_l=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Direct)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Direct), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=423, maxMBSortInHeap=4.219083530284785, sim=ClassicSimilarity, locale=uk, timezone=Australia/NSW
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=201552288,total=511705088
   [junit4]   2> NOTE: All tests run in this JVM: [CdcrReplicationHandlerTest, SpellCheckComponentTest, TestDFRSimilarityFactory, TestNonDefinedSimilarityFactory, TestPKIAuthenticationPlugin, TestSolrQueryResponse, ExplicitHLLTest, DeleteInactiveReplicaTest, ConvertedLegacyTest, TestSolrDynamicMBean, RemoteQueryErrorTest, TestManagedSchema, TestConfigOverlay, HdfsTlogReplayBufferedWhileIndexingTest, LoggingHandlerTest, TestDistributedGrouping, IndexBasedSpellCheckerTest, AliasIntegrationTest, TestRandomCollapseQParserPlugin, SpellingQueryConverterTest, TestFieldResource, TestConfigSets, SpellCheckCollatorTest, SpatialRPTFieldTypeTest, TestLMJelinekMercerSimilarityFactory, DistributedFacetPivotSmallAdvancedTest, TestQuerySenderListener, AsyncMigrateRouteKeyTest, TestRandomRequestDistribution, OpenExchangeRatesOrgProviderTest, TestBlendedInfixSuggestions, HighlighterMaxOffsetTest, TestMacros, RulesTest, HighlighterTest, TimeZoneUtilsTest, TestSchemaVersionResource, TestTrieFacet, ShardSplitTest, TestCodecSupport, TestHighFrequencyDictionaryFactory, TestBinaryResponseWriter, TestSolrDeletionPolicy2, TestTolerantUpdateProcessorRandomCloud, TestFiltering, OpenCloseCoreStressTest, ResponseHeaderTest, DistributedTermsComponentTest, TermVectorComponentTest, TestSchemaNameResource, TestDocumentBuilder, TestSolrQueryParserResource, TestSimpleTrackingShardHandler, SimpleMLTQParserTest, DOMUtilTest, DistributedQueryComponentCustomSortTest, DistanceFunctionTest, SpatialFilterTest, TestHashPartitioner, TestBlobHandler, ExitableDirectoryReaderTest, TestWordDelimiterFilterFactory, SoftAutoCommitTest, SimplePostToolTest, CoreAdminRequestStatusTest, DataDrivenBlockJoinTest, DistributedExpandComponentTest, DistributedMLTComponentTest, DistributedIntervalFacetingTest, ShardRoutingTest, TestEmbeddedSolrServerConstructors, NotRequiredUniqueKeyTest, TestScoreJoinQPScore, TestConfigSetsAPIExclusivity, DocExpirationUpdateProcessorFactoryTest, TestMiniSolrCloudClusterBase, TestPartialUpdateDeduplication, ZkSolrClientTest, MBeansHandlerTest, TestDynamicFieldCollectionResource, PreAnalyzedUpdateProcessorTest, WordBreakSolrSpellCheckerTest, ShardRoutingCustomTest, DateRangeFieldTest, TestLFUCache, HdfsRestartWhileUpdatingTest, SortByFunctionTest, OverseerStatusTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed [490/585 (4!)] on J1 in 3469.61s, 1 test, 1 error <<< FAILURES!

[...truncated 314 lines...]
   [junit4] JVM J1: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/temp/junit4-J1-20160411_101816_005.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid31563.hprof ...
   [junit4] Heap dump file created [713619018 bytes in 59.609 secs]
   [junit4] <<< JVM J1: EOF ----

[...truncated 14 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2363 tests, 2 errors, 3 failures, 31 ignored (16 assumptions) [seed: C7132120DA389C6C]

Total time: 329 minutes 44 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 34 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/34/

1 tests failed.
FAILED:  org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals

Error Message:
this IndexWriter is closed

Stack Trace:
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)
	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)
	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1329)
	at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1312)
	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:140)
	at org.apache.lucene.index.TestDuelingCodecs.createRandomIndex(TestDuelingCodecs.java:139)
	at org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals(TestDuelingCodecsAtNight.java:34)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.apache.lucene.util.fst.FST.<init>(FST.java:387)
	at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsReader.<init>(MemoryPostingsFormat.java:894)
	at org.apache.lucene.codecs.memory.MemoryPostingsFormat.fieldsProducer(MemoryPostingsFormat.java:977)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106)
	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:66)
	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
	at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4032)
	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679)
	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)




Build Log:
[...truncated 1670 lines...]
   [junit4] Suite: org.apache.lucene.index.TestDuelingCodecsAtNight
   [junit4]   2> квіт. 10, 2016 12:28:31 AM com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler uncaughtException
   [junit4]   2> WARNING: Uncaught exception in thread: Thread[Lucene Merge Thread #68,5,TGRP-TestDuelingCodecsAtNight]
   [junit4]   2> org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> 	at __randomizedtesting.SeedInfo.seed([91086C7AB83F8D06]:0)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]   2> Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]   2> 	at org.apache.lucene.util.fst.FST.<init>(FST.java:387)
   [junit4]   2> 	at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsReader.<init>(MemoryPostingsFormat.java:894)
   [junit4]   2> 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat.fieldsProducer(MemoryPostingsFormat.java:977)
   [junit4]   2> 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]   2> 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]   2> 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106)
   [junit4]   2> 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:66)
   [junit4]   2> 	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]   2> 	at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4032)
   [junit4]   2> 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]   2> 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> 
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDuelingCodecsAtNight -Dtests.method=testBigEquals -Dtests.seed=91086C7AB83F8D06 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=uk-UA -Dtests.timezone=PNT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   2264s J0 | TestDuelingCodecsAtNight.testBigEquals <<<
   [junit4]    > Throwable #1: org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:724)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:738)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1329)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1312)
   [junit4]    > 	at org.apache.lucene.index.RandomIndexWriter.addDocument(RandomIndexWriter.java:140)
   [junit4]    > 	at org.apache.lucene.index.TestDuelingCodecs.createRandomIndex(TestDuelingCodecs.java:139)
   [junit4]    > 	at org.apache.lucene.index.TestDuelingCodecsAtNight.testBigEquals(TestDuelingCodecsAtNight.java:34)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at org.apache.lucene.util.fst.FST.<init>(FST.java:387)
   [junit4]    > 	at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsReader.<init>(MemoryPostingsFormat.java:894)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat.fieldsProducer(MemoryPostingsFormat.java:977)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]    > 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106)
   [junit4]    > 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:66)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4032)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)Throwable #2: com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=1636, name=Lucene Merge Thread #68, state=RUNNABLE, group=TGRP-TestDuelingCodecsAtNight]
   [junit4]    > Caused by: org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([91086C7AB83F8D06]:0)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:668)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:648)
   [junit4]    > Caused by: java.lang.OutOfMemoryError: Java heap space
   [junit4]    > 	at org.apache.lucene.util.fst.FST.<init>(FST.java:387)
   [junit4]    > 	at org.apache.lucene.util.fst.FST.<init>(FST.java:313)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat$TermsReader.<init>(MemoryPostingsFormat.java:894)
   [junit4]    > 	at org.apache.lucene.codecs.memory.MemoryPostingsFormat.fieldsProducer(MemoryPostingsFormat.java:977)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:261)
   [junit4]    > 	at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:341)
   [junit4]    > 	at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:106)
   [junit4]    > 	at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:66)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
   [junit4]    > 	at org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:617)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4032)
   [junit4]    > 	at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3679)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:588)
   [junit4]    > 	at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:626)
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/test/J0/temp/lucene.index.TestDuelingCodecsAtNight_91086C7AB83F8D06-001
   [junit4]   2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=6, maxDocsPerChunk=9, blockSize=237), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=6, blockSize=237)), sim=ClassicSimilarity, locale=uk-UA, timezone=PNT
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=68100096,total=525336576
   [junit4]   2> NOTE: All tests run in this JVM: [TestPerSegmentDeletes, TestRegexpRandom, TestDirectMonotonic, TestFixedBitSet, TestOfflineSorter, TestUniqueTermCount, TestMultiCollector, TestDoc, TestPersistentSnapshotDeletionPolicy, TestBytesRef, TestSortRandom, TestLucene50StoredFieldsFormat, TestIsCurrent, TestMergeRateLimiter, TestIndexWriterWithThreads, TestMultiPhraseQuery, TestReaderClosed, TestTermsEnum, Test4GBStoredFields, TestCloseableThreadLocal, TestPolygon, TestCheckIndex, TestMultiValuedNumericRangeQuery, TestSwappedIndexFiles, TestSpanNotQuery, TestWeakIdentityMap, TestDeterminism, TestSynonymQuery, TestBinaryDocument, TestNRTReaderCleanup, TestLSBRadixSorter, TestSpansEnum, TestSegmentInfos, TestQueryRescorer, TestIndexWriterUnicode, TestTragicIndexWriterDeadlock, TestIndexWriterForceMerge, TestBooleanMinShouldMatch, TestSimilarityProvider, TestDocValuesIndexing, TestMergedIterator, TestPhraseQuery, TestTermVectorsWriter, TestSearcherManager, TestFilterLeafReader, TestCustomNorms, TestMultiTermConstantScore, TestPerFieldPostingsFormat, TestCrash, TestSentinelIntSet, TestNumericRangeQuery64, TestTimSorterWorstCase, TestSubScorerFreqs, TestMinimize, TestDuelingCodecsAtNight]
   [junit4] Completed [404/419 (1!)] on J0 in 2292.97s, 3 tests, 1 error <<< FAILURES!

[...truncated 50 lines...]
   [junit4] JVM J0: stdout was not empty, see: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/core/test/temp/junit4-J0-20160410_062104_316.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) ----
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/heapdumps/java_pid1012.hprof ...
   [junit4] Heap dump file created [274962792 bytes in 0.658 secs]
   [junit4] <<< JVM J0: EOF ----

[...truncated 10 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build.xml:50: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 419 suites (1 ignored), 3497 tests, 1 error, 39 ignored (35 assumptions) [seed: 91086C7AB83F8D06]

Total time: 97 minutes 0 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 33 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/33/

6 tests failed.
FAILED:  org.apache.solr.client.solrj.ConnectionReuseTest.test

Error Message:
We expected all communication via streaming client to use one connection! expected=245 got=241

Stack Trace:
java.lang.AssertionError: We expected all communication via streaming client to use one connection! expected=245 got=241
	at __randomizedtesting.SeedInfo.seed([5183B715ACE3B7E2:D9D788CF021FDA1A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.client.solrj.ConnectionReuseTest.test(ConnectionReuseTest.java:150)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:50934/_","node_name":"127.0.0.1:50934__","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf)={   "replicationFactor":"3",   "shards":{"shard1":{       "range":"80000000-7fffffff",       "state":"active",       "replicas":{         "core_node1":{           "state":"down",           "base_url":"http://127.0.0.1:53417/_",           "core":"c8n_1x3_lf_shard1_replica3",           "node_name":"127.0.0.1:53417__"},         "core_node2":{           "core":"c8n_1x3_lf_shard1_replica2",           "base_url":"http://127.0.0.1:50934/_",           "node_name":"127.0.0.1:50934__",           "state":"active",           "leader":"true"},         "core_node3":{           "core":"c8n_1x3_lf_shard1_replica1",           "base_url":"http://127.0.0.1:57768/_",           "node_name":"127.0.0.1:57768__",           "state":"down"}}}},   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",   "autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 1; [core_node2:{"core":"c8n_1x3_lf_shard1_replica2","base_url":"http://127.0.0.1:50934/_","node_name":"127.0.0.1:50934__","state":"active","leader":"true"}]; clusterState: DocCollection(c8n_1x3_lf)={
  "replicationFactor":"3",
  "shards":{"shard1":{
      "range":"80000000-7fffffff",
      "state":"active",
      "replicas":{
        "core_node1":{
          "state":"down",
          "base_url":"http://127.0.0.1:53417/_",
          "core":"c8n_1x3_lf_shard1_replica3",
          "node_name":"127.0.0.1:53417__"},
        "core_node2":{
          "core":"c8n_1x3_lf_shard1_replica2",
          "base_url":"http://127.0.0.1:50934/_",
          "node_name":"127.0.0.1:50934__",
          "state":"active",
          "leader":"true"},
        "core_node3":{
          "core":"c8n_1x3_lf_shard1_replica1",
          "base_url":"http://127.0.0.1:57768/_",
          "node_name":"127.0.0.1:57768__",
          "state":"down"}}}},
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
	at __randomizedtesting.SeedInfo.seed([5183B715ACE3B7E2:D9D788CF021FDA1A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:168)
	at org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Error from server at http://127.0.0.1:39939/ikhy/v/delete_data_dir: Async exception during distributed update: 127.0.0.1:33618 failed to respond

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39939/ikhy/v/delete_data_dir: Async exception during distributed update: 127.0.0.1:33618 failed to respond
	at __randomizedtesting.SeedInfo.seed([5183B715ACE3B7E2:D9D788CF021FDA1A]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:173)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:138)
	at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:152)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:171)
	at org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:99)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=35565, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=35565, name=collection4, state=RUNNABLE, group=TGRP-CollectionsAPIDistributedZkTest]
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:40769: collection already exists: awholynewstresscollection_collection4_0
	at __randomizedtesting.SeedInfo.seed([5183B715ACE3B7E2]:0)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:404)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1192)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:962)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:898)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1593)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1614)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest$1CollectionThread.run(CollectionsAPIDistributedZkTest.java:970)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:53348

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:53348
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:497)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([5183B715ACE3B7E2:3ACC1768D5EC6AD8]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11285 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.StressHdfsTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/init-core-data-001
   [junit4]   2> 51610 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /ikhy/v
   [junit4]   2> 54713 WARN  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 57610 WARN  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 58343 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 58519 WARN  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 58719 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 59003 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_45790_hdfs____v1k17b/webapp
   [junit4]   2> 59261 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 61627 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:45790
   [junit4]   2> 64561 WARN  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 64566 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 64670 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47265_datanode____i514l/webapp
   [junit4]   2> 64957 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 65838 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47265
   [junit4]   2> 66764 WARN  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 66765 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 66794 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_60053_datanode____1jfvzv/webapp
   [junit4]   2> 67316 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 68518 INFO  (SUITE-StressHdfsTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60053
   [junit4]   2> 70105 INFO  (IPC Server handler 4 on 44481) [    ] BlockStateChange BLOCK* processReport: from storage DS-4c3e0f1a-601e-4aec-9da0-9595f0d98d89 node DatanodeRegistration(127.0.0.1, datanodeUuid=6d849f19-ca7d-4a7a-b18f-df2028d62138, infoPort=60053, ipcPort=39527, storageInfo=lv=-56;cid=testClusterID;nsid=293820356;c=0), blocks: 0, hasStaleStorages: true, processing time: 3 msecs
   [junit4]   2> 70105 INFO  (IPC Server handler 5 on 44481) [    ] BlockStateChange BLOCK* processReport: from storage DS-b6d1009d-39cc-4d64-bbd5-6245509b30be node DatanodeRegistration(127.0.0.1, datanodeUuid=0f29ff74-2ec3-45f0-a14f-a5ff39e6d3f0, infoPort=47265, ipcPort=42828, storageInfo=lv=-56;cid=testClusterID;nsid=293820356;c=0), blocks: 0, hasStaleStorages: true, processing time: 0 msecs
   [junit4]   2> 70105 INFO  (IPC Server handler 4 on 44481) [    ] BlockStateChange BLOCK* processReport: from storage DS-27dbb4e8-08e9-414a-8025-a3e429c2359c node DatanodeRegistration(127.0.0.1, datanodeUuid=6d849f19-ca7d-4a7a-b18f-df2028d62138, infoPort=60053, ipcPort=39527, storageInfo=lv=-56;cid=testClusterID;nsid=293820356;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 70110 INFO  (IPC Server handler 5 on 44481) [    ] BlockStateChange BLOCK* processReport: from storage DS-b76d4da9-04f7-4764-baff-feddacdfa88e node DatanodeRegistration(127.0.0.1, datanodeUuid=0f29ff74-2ec3-45f0-a14f-a5ff39e6d3f0, infoPort=47265, ipcPort=42828, storageInfo=lv=-56;cid=testClusterID;nsid=293820356;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 73514 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 73522 INFO  (Thread-238) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 73522 INFO  (Thread-238) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 73750 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.ZkTestServer start zk server on port:51835
   [junit4]   2> 73774 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 74028 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 74084 INFO  (zkCallback-31-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@341c90b6 name:ZooKeeperConnection Watcher:127.0.0.1:51835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74086 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 74086 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 74088 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 74127 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x153fd2997740000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 74148 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 74190 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 74246 INFO  (zkCallback-32-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3309a6b4 name:ZooKeeperConnection Watcher:127.0.0.1:51835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74246 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 74246 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 74249 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 74262 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 74264 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 74266 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 74269 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 74270 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 74275 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 74275 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 74363 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 74363 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 74365 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 74366 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 74367 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 74367 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 74369 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 74369 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 74371 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 74371 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 74373 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 74373 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 74375 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 74375 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 74386 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 74386 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 74388 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 74388 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 74609 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores/collection1
   [junit4]   2> 74634 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 74686 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@19d3718d{/ikhy/v,null,AVAILABLE}
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.e.j.s.ServerConnector Started ServerConnector@4e7a025c{HTTP/1.1,[http/1.1]}{127.0.0.1:33618}
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.e.j.s.Server Started @84229ms
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:44481/hdfs__localhost_44481__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001_tempDir-002_control_data, hostContext=/ikhy/v, hostPort=33618, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores}
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001'
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 74687 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 74718 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 74878 INFO  (zkCallback-33-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@64deb79b name:ZooKeeperConnection Watcher:127.0.0.1:51835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 74879 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 74880 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 75051 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 75051 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/solr.xml
   [junit4]   2> 75066 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores
   [junit4]   2> 75066 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.CoreContainer New CoreContainer 119396957
   [junit4]   2> 75066 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001]
   [junit4]   2> 75066 WARN  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/lib
   [junit4]   2> 75067 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 75072 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 75074 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 75074 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 75074 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:51835/solr
   [junit4]   2> 75117 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 75117 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 75172 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 75208 INFO  (zkCallback-36-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@75884d85 name:ZooKeeperConnection Watcher:127.0.0.1:51835 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75208 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 75209 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 75237 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 75251 INFO  (zkCallback-37-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@b3fd0b5 name:ZooKeeperConnection Watcher:127.0.0.1:51835/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 75251 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 75266 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 75278 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 75310 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 75325 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 75334 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 75407 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 75410 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 75411 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 75413 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 75414 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 75478 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 75534 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 75537 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 75541 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 75545 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000
   [junit4]   2> 75546 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:33618_ikhy%2Fv
   [junit4]   2> 75546 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 75552 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.Overseer Overseer (id=95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) starting
   [junit4]   2> 75572 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 75789 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 75805 INFO  (OverseerStateUpdate-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 75805 INFO  (OverseerCollectionConfigSetProcessor-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 76010 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33618_ikhy%2Fv
   [junit4]   2> 76014 INFO  (OverseerStateUpdate-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 76023 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:33618_ikhy%2Fv
   [junit4]   2> 76029 INFO  (zkCallback-37-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 76029 INFO  (zkCallback-37-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 76315 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 76315 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 76320 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores
   [junit4]   2> 76322 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 76322 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores/collection1
   [junit4]   2> 76322 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 76663 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 76663 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 76802 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 76802 INFO  (TEST-StressHdfsTest.test-seed#[5183B715ACE3B7E2]) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 76805 INFO  (OverseerStateUpdate-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33618/ikhy/v",
   [junit4]   2>   "node_name":"127.0.0.1:33618_ikhy%2Fv",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 76812 INFO  (OverseerStateUpdate-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:33618/ikhy/v",
   [junit4]   2>   "node_name":"127.0.0.1:33618_ikhy%2Fv",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 76814 INFO  (OverseerStateUpdate-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 76831 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 76833 INFO  (OverseerStateUpdate-95698372181360644-127.0.0.1:33618_ikhy%2Fv-n_0000000000) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 76890 INFO  (zkCallback-37-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 77874 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 77874 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 77875 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 77875 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 77876 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 77876 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores/collection1'
   [junit4]   2> 77876 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 77876 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 77890 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 78105 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 78108 WARN  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 78109 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 78170 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 78208 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 78591 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 78616 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 78617 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 78674 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 78676 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 78682 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 78682 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 78684 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 78684 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 78684 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 78709 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:44481/solr_hdfs_home
   [junit4]   2> 78709 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 78709 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.StressHdfsTest_5183B715ACE3B7E2-001/control-001/cores/collection1], dataDir=[hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 78709 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@42dcbe83
   [junit4]   2> 78722 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 79309 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 79327 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 79526 WARN  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:44481/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 79526 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 79536 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 79536 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 79537 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new global HDFS BlockCache
   [junit4]   2> 83884 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 83884 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 83884 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=15, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1465111180179477]
   [junit4]   2> 85390 INFO  (IPC Server handler 9 on 44481) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59273 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4c3e0f1a-601e-4aec-9da0-9595f0d98d89:NORMAL:127.0.0.1:59273|RBW], ReplicaUnderConstruction[[DISK]DS-b6d1009d-39cc-4d64-bbd5-6245509b30be:NORMAL:127.0.0.1:59003|RBW]]} size 71
   [junit4]   2> 85394 INFO  (IPC Server handler 0 on 44481) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:59003 is added to blk_1073741825_1001 size 71
   [junit4]   2> 85790 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@72ee4eca); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 85790 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 85843 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 85852 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 85861 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 85862 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 85862 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 85862 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 85862 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 85864 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 85865 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 85865 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 85867 WARN  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 85868 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 85871 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 85872 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 85880 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 85940 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 85940 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 85944 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 85944 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 85944 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 86230 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 86230 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 86232 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=595380499, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 86536 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:44481/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@72ee4eca); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 86537 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 86537 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@13f47306[collection1] main]
   [junit4]   2> 86537 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 86562 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 86562 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 86578 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 86578 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 86578 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 86578 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 86579 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 86579 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 86581 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 86591 INFO  (searcherExecutor-198-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@13f47306[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 86595 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 86595 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 86595 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 86595 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1531173968287891456
   [junit4]   2> 86596 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1531173968287891456
   [junit4]   2> 86614 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 86622 INFO  (coreLoadExecutor-197-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 86642 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:33618/ikhy/v collection:control_collection shard:shard1
   [junit4]   2> 86656 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 86698 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95698372181360644-core_node1-n_0000000000
   [junit4]   2> 86698 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=180000
   [junit4]   2> 86702 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 86702 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 86702 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:33618/ikhy/v/collection1/
   [junit4]   2> 86702 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 86702 INFO  (coreZkRegister-191-thread-1-processing-n:127.0.0.1:33618_ikhy%2Fv x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:33618_ikhy%2Fv c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:33618/ikhy/v/collection1/ has no replicas
   [junit4]   2> 86703 INFO  (coreZkRegister-191-thread-1-processing-n:12

[...truncated too long message...]

)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Caused by: java.lang.NullPointerException
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlocksMap.size(BlocksMap.java:198)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.getTotalBlocks(BlockManager.java:3291)
   [junit4]   2> 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlocksTotal(FSNamesystem.java:6223)
   [junit4]   2> 	... 54 more
   [junit4]   2> 5690237 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[5183B715ACE3B7E2]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_5183B715ACE3B7E2-001
   [junit4]   2> Apr 10, 2016 12:08:11 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=PostingsFormat(name=LuceneVarGapFixedInterval), range_facet_l_dv=PostingsFormat(name=Direct), _version_=PostingsFormat(name=LuceneVarGapFixedInterval), multiDefault=PostingsFormat(name=Memory doPackFST= true), a_t=PostingsFormat(name=Memory doPackFST= true), intDefault=PostingsFormat(name=LuceneVarGapFixedInterval), a_i1=PostingsFormat(name=Memory doPackFST= true), id=PostingsFormat(name=Direct), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= true), text=PostingsFormat(name=Asserting), range_facet_l=PostingsFormat(name=Memory doPackFST= true), timestamp=PostingsFormat(name=Memory doPackFST= true)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=945, maxMBSortInHeap=4.17361962605737, sim=RandomSimilarity(queryNorm=false,coord=no): {}, locale=he-IL, timezone=PRC
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=86320160,total=489684992
   [junit4]   2> NOTE: All tests run in this JVM: [BigEndianAscendingWordDeserializerTest, MoreLikeThisHandlerTest, TestNonDefinedSimilarityFactory, TestSolrDeletionPolicy2, TestDocumentBuilder, TestExactSharedStatsCache, SpellCheckCollatorTest, TestNoOpRegenerator, StressHdfsTest, TestReloadAndDeleteDocs, AtomicUpdatesTest, TestFieldSortValues, TestRecoveryHdfs, ConvertedLegacyTest, DocValuesTest, DistributedMLTComponentTest, ConfigSetsAPITest, TestFieldResource, OpenCloseCoreStressTest, TestReqParamsAPI, DocExpirationUpdateProcessorFactoryTest, UpdateRequestProcessorFactoryTest, DeleteShardTest, TestManagedResourceStorage, TestTrieFacet, DistanceFunctionTest, TestSurroundQueryParser, ClusterStateUpdateTest, TestQuerySenderNoQuery, TestCursorMarkWithoutUniqueKey, TestBM25SimilarityFactory, TestRTimerTree, TestFiltering, SimplePostToolTest, TestSchemaNameResource, SoftAutoCommitTest, HighlighterTest, TestConfigOverlay, SortSpecParsingTest, TestQuerySenderListener, DistributedExpandComponentTest, DeleteInactiveReplicaTest, ShardRoutingTest, NotRequiredUniqueKeyTest, CursorMarkTest, SpatialFilterTest, TestMacros, ExplicitHLLTest, TestTolerantUpdateProcessorRandomCloud, RollingRestartTest, TestPartialUpdateDeduplication, TestScoreJoinQPScore, CSVRequestHandlerTest, TermVectorComponentTest, TestHighFrequencyDictionaryFactory, TestPKIAuthenticationPlugin, MinimalSchemaTest, TestRandomCollapseQParserPlugin, TestLeaderElectionZkExpiry, TestCoreContainer, MigrateRouteKeyTest, CoreAdminRequestStatusTest, TestQueryUtils, LeaderFailoverAfterPartitionTest, ExitableDirectoryReaderTest, SpellPossibilityIteratorTest, TestCloudPivotFacet, ChaosMonkeySafeLeaderTest, DistributedQueryComponentCustomSortTest, SpellCheckComponentTest, TestQueryTypes, SimpleCollectionCreateDeleteTest, SolrInfoMBeanTest, PreAnalyzedUpdateProcessorTest, TestSimpleQParserPlugin, DistributedFacetPivotSmallAdvancedTest, NumericFieldsTest, MultiTermTest, FacetPivotSmallTest, TestEmbeddedSolrServerConstructors, BJQParserTest, TestConfigSets, CollectionsAPIDistributedZkTest, HdfsNNFailoverTest, TermsComponentTest, SchemaVersionSpecificBehaviorTest, TestUpdate, BasicAuthIntegrationTest, AsyncMigrateRouteKeyTest, BasicFunctionalityTest, DateRangeFieldTest, AutoCommitTest, TestDistributedGrouping, TestFaceting, TestHashPartitioner, TermVectorComponentDistributedTest, ZkControllerTest, TestJoin, TestStressVersions, HardAutoCommitTest, DistributedTermsComponentTest, SolrCoreTest, SolrCmdDistributorTest, QueryElevationComponentTest, DirectUpdateHandlerTest, TestIndexSearcher, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, SolrIndexSplitterTest, TestCoreDiscovery, SignatureUpdateProcessorFactoryTest, SuggesterFSTTest, SolrRequestParserTest, TestFoldingMultitermQuery, SuggesterTSTTest, SuggesterTest, TestStressLucene, TestTrie, SuggesterWFSTTest, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestPseudoReturnFields, TestWordDelimiterFilterFactory, QueryEqualityTest, SortByFunctionTest, CacheHeaderTest, IndexBasedSpellCheckerTest, DisMaxRequestHandlerTest, StandardRequestHandlerTest, DirectSolrSpellCheckerTest, TestReversedWildcardFilterFactory, PrimitiveFieldTypeTest, TestOmitPositions, FileBasedSpellCheckerTest, XmlUpdateRequestHandlerTest, RequestHandlersTest, DocumentBuilderTest, PathHierarchyTokenizerFactoryTest, TestIndexingPerformance, RequiredFieldsTest, FastVectorHighlighterTest, LoggingHandlerTest, RegexBoostProcessorTest, SolrPluginUtilsTest, IndexSchemaTest, TestJmxIntegration, TestCollationField, TestCSVResponseWriter, MBeansHandlerTest, TestAnalyzedSuggestions, UniqFieldsUpdateProcessorFactoryTest, PingRequestHandlerTest, SearchHandlerTest, TestBinaryResponseWriter, TestLFUCache, HighlighterConfigTest, AlternateDirectoryTest, TestSolrIndexConfig, ResponseLogComponentTest, SolrIndexConfigTest, BadComponentTest, TestStressRecovery, PreAnalyzedFieldTest, SpellingQueryConverterTest, RAMDirectoryFactoryTest, TestSolrJ, SliceStateTest, FileUtilsTest, TestDistributedMissingSort, TestSimpleTrackingShardHandler, ConnectionReuseTest, CdcrRequestHandlerTest, CollectionsAPIAsyncDistributedZkTest, ConnectionManagerTest, DistributedQueueTest, DistributedVersionInfoTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerRolesTest, OverseerStatusTest, OverseerTaskQueueTest, RemoteQueryErrorTest, SSLMigrationTest, SaslZkACLProviderTest, SharedFSAutoReplicaFailoverTest, TestCryptoKeys, TestLeaderInitiatedRecoveryThread, TestMiniSolrCloudClusterBase, TestRandomRequestDistribution, TestReplicaProperties, TestShortCircuitedRequests, TestSolrCloudWithKerberosAlt, TestTolerantUpdateProcessorCloud, VMParamsZkACLAndCredentialsProvidersTest, HdfsBasicDistributedZk2Test, HdfsChaosMonkeyNothingIsSafeTest, HdfsChaosMonkeySafeLeaderTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [477/585 (6!)] on J1 in 607.99s, 1 test, 1 error <<< FAILURES!

[...truncated 357 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2363 tests, 4 errors, 2 failures, 31 ignored (16 assumptions) [seed: 5183B715ACE3B7E2]

Total time: 299 minutes 19 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 32 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/32/

5 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:41346/o_d/nz

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:41346/o_d/nz
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.deletePartiallyCreatedCollection(CollectionsAPIDistributedZkTest.java:243)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:171)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
There were too many update fails (83 > 60) - we expect it can happen, but shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails (83 > 60) - we expect it can happen, but shouldn't easily
	at __randomizedtesting.SeedInfo.seed([D16D03BF5A4DBE22:59393C65F4B1D3DA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:218)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:39334/_pbi/is

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:39334/_pbi/is
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:497)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:495)
	... 47 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest.test

Error Message:
There are still nodes recoverying - waited for 440 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 440 seconds
	at __randomizedtesting.SeedInfo.seed([D16D03BF5A4DBE22:59393C65F4B1D3DA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1412)
	at org.apache.solr.cloud.TlogReplayBufferedWhileIndexingTest.test(TlogReplayBufferedWhileIndexingTest.java:99)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([D16D03BF5A4DBE22:BA22A3C223426318]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11833 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/init-core-data-001
   [junit4]   2> 1367398 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /y_yh/yq
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 1367517 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 1367529 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1367536 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1367578 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_50178_hdfs____w7z669/webapp
   [junit4]   2> 1367676 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1368022 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:50178
   [junit4]   2> 1368238 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1368239 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1368248 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47245_datanode____h84s6v/webapp
   [junit4]   2> 1368363 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1368783 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47245
   [junit4]   2> 1368818 WARN  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 1368819 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 1368830 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_54900_datanode____d1lfxd/webapp
   [junit4]   2> 1368988 INFO  (IPC Server handler 3 on 42133) [    ] BlockStateChange BLOCK* processReport: from storage DS-609b957a-eb87-4b86-a516-64c985034ef8 node DatanodeRegistration(127.0.0.1, datanodeUuid=d87b82cb-19a0-4f74-8d31-601b97ff74b9, infoPort=47245, ipcPort=55194, storageInfo=lv=-56;cid=testClusterID;nsid=1621610907;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 1368988 INFO  (IPC Server handler 3 on 42133) [    ] BlockStateChange BLOCK* processReport: from storage DS-0ca90e83-b8d5-4e90-ba56-68c7e541c936 node DatanodeRegistration(127.0.0.1, datanodeUuid=d87b82cb-19a0-4f74-8d31-601b97ff74b9, infoPort=47245, ipcPort=55194, storageInfo=lv=-56;cid=testClusterID;nsid=1621610907;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1369077 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 1369502 INFO  (SUITE-HdfsTlogReplayBufferedWhileIndexingTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:54900
   [junit4]   2> 1369780 INFO  (IPC Server handler 7 on 42133) [    ] BlockStateChange BLOCK* processReport: from storage DS-0bc88fb3-df07-4c93-8a78-f4f729b6c52c node DatanodeRegistration(127.0.0.1, datanodeUuid=b219c554-45c5-440e-9567-24b80284e021, infoPort=54900, ipcPort=47114, storageInfo=lv=-56;cid=testClusterID;nsid=1621610907;c=0), blocks: 0, hasStaleStorages: true, processing time: 2 msecs
   [junit4]   2> 1369786 INFO  (IPC Server handler 7 on 42133) [    ] BlockStateChange BLOCK* processReport: from storage DS-753c5a94-f012-4d77-b2fe-45a31dff256a node DatanodeRegistration(127.0.0.1, datanodeUuid=b219c554-45c5-440e-9567-24b80284e021, infoPort=54900, ipcPort=47114, storageInfo=lv=-56;cid=testClusterID;nsid=1621610907;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 1369988 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1369988 INFO  (Thread-7826) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1369988 INFO  (Thread-7826) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1370090 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.ZkTestServer start zk server on port:48772
   [junit4]   2> 1370090 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1370102 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1370124 INFO  (zkCallback-599-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@fd9a771 name:ZooKeeperConnection Watcher:127.0.0.1:48772 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1370133 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1370133 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1370133 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 1370172 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1370172 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1370180 INFO  (zkCallback-600-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@68f02696 name:ZooKeeperConnection Watcher:127.0.0.1:48772/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1370180 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1370180 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1370180 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 1370181 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 1370181 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 1370182 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 1370183 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1370183 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 1370184 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 1370184 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 1370185 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1370185 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1370186 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1370186 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 1370187 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1370187 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 1370188 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1370188 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 1370189 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1370189 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 1370190 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1370190 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1370191 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1370191 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1370191 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1370192 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 1370192 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1370192 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 1370250 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores/collection1
   [junit4]   2> 1370251 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9fa3506{/y_yh/yq,null,AVAILABLE}
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.e.j.s.ServerConnector Started ServerConnector@282ca27b{HTTP/1.1,[http/1.1]}{127.0.0.1:50424}
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.e.j.s.Server Started @1377419ms
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:42133/hdfs__localhost_42133__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J1_temp_solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001_tempDir-002_control_data, hostContext=/y_yh/yq, hostPort=50424, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores}
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001'
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1370335 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1370336 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1370346 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1370358 INFO  (zkCallback-601-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19688f22 name:ZooKeeperConnection Watcher:127.0.0.1:48772/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1370362 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1370362 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1370371 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1370372 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/solr.xml
   [junit4]   2> 1370383 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores
   [junit4]   2> 1370383 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.CoreContainer New CoreContainer 929334778
   [junit4]   2> 1370383 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001]
   [junit4]   2> 1370384 WARN  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/lib
   [junit4]   2> 1370384 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 1370385 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1370385 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 1370385 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 1370385 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48772/solr
   [junit4]   2> 1370385 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 1370385 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 1370394 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1370414 INFO  (zkCallback-604-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@317e9f04 name:ZooKeeperConnection Watcher:127.0.0.1:48772 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1370421 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1370421 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 1370447 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 1370461 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7dd991c3 name:ZooKeeperConnection Watcher:127.0.0.1:48772/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 1370461 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 1370472 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 1370578 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 1370614 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 1370621 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 1370623 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 1370624 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 1370625 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 1370626 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 1370626 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 1370627 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 1370627 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1370628 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 1370628 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 1370629 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 1370629 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000
   [junit4]   2> 1370629 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:50424_y_yh%2Fyq
   [junit4]   2> 1370629 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 1370630 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.Overseer Overseer (id=95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) starting
   [junit4]   2> 1370631 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 1370633 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 1370634 INFO  (OverseerStateUpdate-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 1370634 INFO  (OverseerCollectionConfigSetProcessor-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 1370642 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50424_y_yh%2Fyq
   [junit4]   2> 1370643 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:50424_y_yh%2Fyq
   [junit4]   2> 1370643 INFO  (OverseerStateUpdate-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 1370645 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 1370648 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1370651 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 1370651 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 1370652 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores
   [junit4]   2> 1370652 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 1370653 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores/collection1
   [junit4]   2> 1370653 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 1370654 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1370654 INFO  (TEST-HdfsTlogReplayBufferedWhileIndexingTest.test-seed#[D16D03BF5A4DBE22]) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 1370655 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 1370655 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 1370655 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 1370671 INFO  (OverseerStateUpdate-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50424/y_yh/yq",
   [junit4]   2>   "node_name":"127.0.0.1:50424_y_yh%2Fyq",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 1370678 INFO  (OverseerStateUpdate-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:50424/y_yh/yq",
   [junit4]   2>   "node_name":"127.0.0.1:50424_y_yh%2Fyq",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 1370686 INFO  (OverseerStateUpdate-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 1370694 INFO  (OverseerStateUpdate-95693743516286980-127.0.0.1:50424_y_yh%2Fyq-n_0000000000) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1370714 INFO  (zkCallback-605-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 1371656 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 1371656 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 1371656 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 1371656 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1371656 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1371657 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores/collection1'
   [junit4]   2> 1371657 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 1371657 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 1371658 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 1371661 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 1371664 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 1371671 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 1371675 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1371725 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 1371725 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 1371726 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1371728 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 1371777 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1
   [junit4]   2> 1371777 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1371777 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 1371778 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:42133/solr_hdfs_home
   [junit4]   2> 1371778 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 1371778 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.hdfs.HdfsTlogReplayBufferedWhileIndexingTest_D16D03BF5A4DBE22-001/control-001/cores/collection1], dataDir=[hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 1371778 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@188ae9e
   [junit4]   2> 1371786 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1371827 WARN  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.h.HdfsDirectory The NameNode is in SafeMode - Solr will wait 5 seconds and try again.
   [junit4]   2> 1376840 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 1376841 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 1376862 WARN  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:42133/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 1376862 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1376878 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 1376879 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [16777216] will allocate [1] slabs and use ~[16777216] bytes
   [junit4]   2> 1376902 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 1376902 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 1376903 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=47, maxMergeAtOnceExplicit=21, maxMergedSegmentMB=19.99609375, floorSegmentMB=1.220703125, forceMergeDeletesPctAllowed=14.933082781567812, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.41621666154360504
   [junit4]   2> 1377104 INFO  (IPC Server handler 5 on 42133) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:36236 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-609b957a-eb87-4b86-a516-64c985034ef8:NORMAL:127.0.0.1:56411|RBW], ReplicaUnderConstruction[[DISK]DS-753c5a94-f012-4d77-b2fe-45a31dff256a:NORMAL:127.0.0.1:36236|RBW]]} size 0
   [junit4]   2> 1377111 INFO  (IPC Server handler 4 on 42133) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:56411 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-753c5a94-f012-4d77-b2fe-45a31dff256a:NORMAL:127.0.0.1:36236|RBW], ReplicaUnderConstruction[[DISK]DS-0ca90e83-b8d5-4e90-ba56-68c7e541c936:NORMAL:127.0.0.1:56411|FINALIZED]]} size 0
   [junit4]   2> 1377114 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ea33787); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1377114 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 1377119 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 1377120 WARN  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 1377121 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1377121 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1377122 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1377122 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 1377275 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 1377278 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 1377278 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 1377282 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 1377282 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1377282 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 1377296 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommited for 10000ms; 
   [junit4]   2> 1377296 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: if uncommited for 3000ms; 
   [junit4]   2> 1377297 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.MockRandomMergePolicy: org.apache.lucene.index.MockRandomMergePolicy@2dc175a3
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:42133/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@5ea33787); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@dd949a3[collection1] main]
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 1377305 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1377306 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 1377306 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1377306 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 1377306 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1377306 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 1377306 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1377310 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 1377311 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 1377311 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 1377311 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1531099903938265088
   [junit4]   2> 1377312 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1531099903938265088
   [junit4]   2> 1377312 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 1377313 INFO  (searcherExecutor-1725-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@dd949a3[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1377313 INFO  (coreLoadExecutor-1724-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 1377319 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:50424/y_yh/yq collection:control_collection shard:shard1
   [junit4]   2> 1377320 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 1377322 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.LeaderElector Joined leadership election with path: /collections/control_collection/leader_elect/shard1/election/95693743516286980-core_node1-n_0000000000
   [junit4]   2> 1377322 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Running the leader process for shard=shard1 and weAreReplacement=false and leaderVoteWait=300000
   [junit4]   2> 1377323 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1377323 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1377323 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:50424_y_yh%2Fyq c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:50424/y_yh/yq/collection1/
   [junit4]   2> 1377323 INFO  (coreZkRegister-1718-thread-1-processing-n:127.0.0.1:50424_y_yh%2Fyq x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:5

[...truncated too long message...]

c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 5569418 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 5569419 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 5569419 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 5569420 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 5569421 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 5569421 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 5569422 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 5569422 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 5569422 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 5569423 INFO  (zkCallback-3795-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 5569424 INFO  (zkCallback-3795-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 5569478 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 5569526 INFO  (zkCallback-3795-thread-1) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 5569529 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51486 51486
   [junit4]   2> 5569830 INFO  (Thread-83283) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:51486 51486
   [junit4]   2> 5569836 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[D16D03BF5A4DBE22]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 5569837 INFO  (SUITE-ZkStateReaderTest-seed#[D16D03BF5A4DBE22]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.cloud.overseer.ZkStateReaderTest_D16D03BF5A4DBE22-001
   [junit4]   2> Apr 09, 2016 4:07:26 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=1357, maxMBSortInHeap=5.15693007768448, sim=RandomSimilarity(queryNorm=false,coord=yes): {}, locale=ar-TN, timezone=America/Inuvik
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=304639336,total=493879296
   [junit4]   2> NOTE: All tests run in this JVM: [RollingRestartTest, DocValuesTest, TestLFUCache, SortByFunctionTest, SecurityConfHandlerTest, TestSolrFieldCacheMBean, TestMacros, OverriddenZkACLAndCredentialsProvidersTest, AliasIntegrationTest, TestBlobHandler, TestRequestForwarding, TestRecoveryHdfs, SpatialFilterTest, MinimalSchemaTest, OpenExchangeRatesOrgProviderTest, ChaosMonkeyNothingIsSafeTest, LeaderFailoverAfterPartitionTest, TestExactSharedStatsCache, TestGeoJSONResponseWriter, IndexBasedSpellCheckerTest, AddBlockUpdateTest, TestPKIAuthenticationPlugin, DistributedIntervalFacetingTest, TestHdfsUpdateLog, TestSchemaNameResource, ZkStateWriterTest, PreAnalyzedUpdateProcessorTest, SpellPossibilityIteratorTest, AtomicUpdatesTest, WordBreakSolrSpellCheckerTest, BigEndianAscendingWordDeserializerTest, TestUpdate, TestRandomRequestDistribution, TermVectorComponentTest, HighlighterTest, DirectSolrSpellCheckerTest, DistributedMLTComponentTest, HdfsTlogReplayBufferedWhileIndexingTest, MoreLikeThisHandlerTest, ZkSolrClientTest, SoftAutoCommitTest, SpellCheckComponentTest, ConfigSetsAPITest, TestHashPartitioner, HdfsChaosMonkeyNothingIsSafeTest, ShardSplitTest, TestSolrQueryParserResource, HdfsRestartWhileUpdatingTest, TestQueryTypes, TestBinaryResponseWriter, DocumentBuilderTest, FastVectorHighlighterTest, TestConfigSetsAPIExclusivity, TestTrie, CSVRequestHandlerTest, DOMUtilTest, CoreAdminRequestStatusTest, TestFiltering, SolrPluginUtilsTest, TestManagedResourceStorage, TestSurroundQueryParser, TestCursorMarkWithoutUniqueKey, TestReloadAndDeleteDocs, DistributedTermsComponentTest, TestSizeLimitedDistributedMap, TestSolrDeletionPolicy2, SolrInfoMBeanTest, NotRequiredUniqueKeyTest, TestBlendedInfixSuggestions, UpdateRequestProcessorFactoryTest, DateRangeFieldTest, BJQParserTest, TestScoreJoinQPScore, HighlighterMaxOffsetTest, TestBM25SimilarityFactory, SimpleCollectionCreateDeleteTest, DistribDocExpirationUpdateProcessorTest, TestInfoStreamLogging, StressHdfsTest, UUIDUpdateProcessorFallbackTest, CursorMarkTest, SpellingQueryConverterTest, BlockJoinFacetSimpleTest, TestCollapseQParserPlugin, DistributedFacetPivotSmallAdvancedTest, TestWordDelimiterFilterFactory, TestSimpleQParserPlugin, ChaosMonkeySafeLeaderTest, TestNonDefinedSimilarityFactory, TestFieldSortValues, RulesTest, TestPivotHelperCode, TestDistributedGrouping, TestFaceting, TestRecovery, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, TestJoin, TestStressVersions, SolrCoreTest, PeerSyncTest, TestFunctionQuery, BasicFunctionalityTest, TestLazyCores, ShowFileRequestHandlerTest, DistributedQueryElevationComponentTest, AnalysisAfterCoreReloadTest, SpellCheckCollatorTest, SolrRequestParserTest, TestStressLucene, SchemaVersionSpecificBehaviorTest, TestSolrDeletionPolicy1, TestQueryUtils, StandardRequestHandlerTest, TermsComponentTest, RequiredFieldsTest, IndexSchemaRuntimeFieldTest, IndexSchemaTest, TestCSVResponseWriter, MBeansHandlerTest, TestAnalyzedSuggestions, TestPartialUpdateDeduplication, TestQuerySenderListener, SolrIndexConfigTest, MultiTermTest, SampleTest, TestBinaryField, NumericFieldsTest, TestElisionMultitermQuery, TestConfig, TestFuzzyAnalyzedSuggestions, TestSolrCoreProperties, TestLuceneMatchVersion, TestPhraseSuggestions, TestCodecSupport, TestXIncludeConfig, TestDFRSimilarityFactory, TestPerFieldSimilarity, TestLMJelinekMercerSimilarityFactory, TestFastWriter, TestJmxMonitoredMap, TimeZoneUtilsTest, ResourceLoaderTest, TestFastOutputStream, ScriptEngineTest, TestFastLRUCache, DateMathParserTest, PreAnalyzedFieldTest, TestSystemIdResolver, PrimUtilsTest, TestSolrJ, TestUtils, TestDocumentBuilder, ZkNodePropsTest, SliceStateTest, SystemInfoHandlerTest, FileUtilsTest, CircularListTest, TestRTGBase, CursorPagingTest, TestJettySolrRunner, AssignTest, AsyncCallRequestStatusResponseTest, CdcrReplicationHandlerTest, CollectionsAPIAsyncDistributedZkTest, CollectionsAPISolrJTest, DeleteInactiveReplicaTest, DeleteShardTest, DistribJoinFromCollectionTest, DistributedVersionInfoTest, HttpPartitionTest, MultiThreadedOCPTest, OverseerStatusTest, RecoveryAfterSoftCommitTest, ReplicationFactorTest, SaslZkACLProviderTest, TestCloudDeleteByQuery, TestCloudInspectUtil, TestCloudPivotFacet, TestCollectionAPI, TestConfigSetsAPI, TestDistribDocBasedVersion, TestLeaderInitiatedRecoveryThread, TestTolerantUpdateProcessorRandomCloud, HdfsCollectionsAPIDistributedZkTest, HdfsNNFailoverTest, HdfsRecoverLeaseTest, HdfsSyncSliceTest, TestClusterStateMutator, ZkStateReaderTest]
   [junit4] Completed [479/585 (5!)] on J1 in 3.20s, 5 tests, 1 error <<< FAILURES!

[...truncated 365 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2363 tests, 3 errors, 2 failures, 31 ignored (16 assumptions) [seed: D16D03BF5A4DBE22]

Total time: 296 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 31 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/31/

2 tests failed.
FAILED:  org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat.testBinaryVariableLengthVsStoredFields

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([613E7B5CBABF0707]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([613E7B5CBABF0707]:0)




Build Log:
[...truncated 6656 lines...]
   [junit4] Suite: org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat
   [junit4]   2> apr 07, 2016 5:07:52 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat
   [junit4]   2>    1) Thread[id=12, name=TEST-TestMemoryDocValuesFormat.testBinaryVariableLengthVsStoredFields-seed#[613E7B5CBABF0707], state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4328)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1802)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1736)
   [junit4]   2>         at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryVsStoredFields(BaseDocValuesFormatTestCase.java:1392)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.testBinaryVariableLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1419)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2>    2) Thread[id=1, name=main, state=WAITING, group=main]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1245)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1319)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:601)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:450)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:243)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:354)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:10)
   [junit4]   2>    3) Thread[id=9, name=JUnit4-serializer-daemon, state=TIMED_WAITING, group=main]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:47)
   [junit4]   2>    4) Thread[id=11, name=SUITE-TestMemoryDocValuesFormat-seed#[613E7B5CBABF0707], state=RUNNABLE, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Thread.getStackTrace(Thread.java:1552)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:688)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$4.run(ThreadLeakControl.java:685)
   [junit4]   2>         at java.security.AccessController.doPrivileged(Native Method)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getStackTrace(ThreadLeakControl.java:685)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.getThreadsWithTraces(ThreadLeakControl.java:701)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:64)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:414)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:681)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:140)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:591)
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestMemoryDocValuesFormat -Dtests.method=testBinaryVariableLengthVsStoredFields -Dtests.seed=613E7B5CBABF0707 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sk-SK -Dtests.timezone=America/Jujuy -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   6864s J2 | TestMemoryDocValuesFormat.testBinaryVariableLengthVsStoredFields <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([613E7B5CBABF0707]:0)
   [junit4]   2> apr 07, 2016 5:07:52 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> apr 07, 2016 5:08:12 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.lucene.codecs.memory.TestMemoryDocValuesFormat: 
   [junit4]   2>    1) Thread[id=12, name=TEST-TestMemoryDocValuesFormat.testBinaryVariableLengthVsStoredFields-seed#[613E7B5CBABF0707], state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.doWait(IndexWriter.java:4328)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1802)
   [junit4]   2>         at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1736)
   [junit4]   2>         at org.apache.lucene.index.RandomIndexWriter.forceMerge(RandomIndexWriter.java:421)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.doTestBinaryVsStoredFields(BaseDocValuesFormatTestCase.java:1392)
   [junit4]   2>         at org.apache.lucene.index.BaseDocValuesFormatTestCase.testBinaryVariableLengthVsStoredFields(BaseDocValuesFormatTestCase.java:1419)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> apr 07, 2016 5:08:12 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=12, name=TEST-TestMemoryDocValuesFormat.testBinaryVariableLengthVsStoredFields-seed#[613E7B5CBABF0707], state=TIMED_WAITING, group=TGRP-TestMemoryDocValuesFormat]
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build/codecs/test/J2/temp/lucene.codecs.memory.TestMemoryDocValuesFormat_613E7B5CBABF0707-001
   [junit4]   2> Apr 07, 2016 8:08:12 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: All leaked threads terminated.
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {}, docValues:{}, maxPointsInLeafNode=1884, maxMBSortInHeap=5.173936306470952, sim=ClassicSimilarity, locale=sk-SK, timezone=America/Jujuy
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=176956984,total=300417024
   [junit4]   2> NOTE: All tests run in this JVM: [TestMemoryDocValuesFormat]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestMemoryDocValuesFormat -Dtests.seed=613E7B5CBABF0707 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=sk-SK -Dtests.timezone=America/Jujuy -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | TestMemoryDocValuesFormat (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([613E7B5CBABF0707]:0)
   [junit4] Completed [18/22 (1!)] on J2 in 7220.88s, 94 tests, 2 errors <<< FAILURES!

[...truncated 27 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/build.xml:476: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:2187: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/module-build.xml:58: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 22 suites, 658 tests, 1 suite-level error, 1 error, 3 ignored (3 assumptions) [seed: 613E7B5CBABF0707]

Total time: 253 minutes 20 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 30 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/30/

5 tests failed.
FAILED:  org.apache.solr.cloud.TestLeaderInitiatedRecoveryThread.testPublishDownState

Error Message:
expected:<27> but was:<28>

Stack Trace:
java.lang.AssertionError: expected:<27> but was:<28>
	at __randomizedtesting.SeedInfo.seed([E702D057C9439CB2:B97F72A96D5ED54F]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.junit.Assert.assertEquals(Assert.java:456)
	at org.apache.solr.cloud.TestLeaderInitiatedRecoveryThread.testPublishDownState(TestLeaderInitiatedRecoveryThread.java:114)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestTolerantUpdateProcessorCloud

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([E702D057C9439CB2]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorCloud.java:129)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
[/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_E702D057C9439CB2-001/solr-instance-008/./collection1/data, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_E702D057C9439CB2-001/solr-instance-008/./collection1/data/index.20160406105801441, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_E702D057C9439CB2-001/solr-instance-008/./collection1/data/index.20160406105801245] expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_E702D057C9439CB2-001/solr-instance-008/./collection1/data, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_E702D057C9439CB2-001/solr-instance-008/./collection1/data/index.20160406105801441, /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J1/temp/solr.handler.TestReplicationHandler_E702D057C9439CB2-001/solr-instance-008/./collection1/data/index.20160406105801245] expected:<2> but was:<3>
	at __randomizedtesting.SeedInfo.seed([E702D057C9439CB2:10713E0F0FAB3354]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.handler.TestReplicationHandler.checkForSingleIndex(TestReplicationHandler.java:818)
	at org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1248)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:34519/be

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:34519/be
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:583)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:528)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:477)
	... 47 more


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
replicaCount expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: replicaCount expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([E702D057C9439CB2:6F56EF8D67BFF14A]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testNoConfigSetExist(CollectionsAPIDistributedZkTest.java:601)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:166)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11290 lines...]
   [junit4] Suite: org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/init-core-data-001
   [junit4]   2> 315865 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false)
   [junit4]   2> 315865 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 316022 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 316032 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 316041 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 316119 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_56969_hdfs____.69jt23/webapp
   [junit4]   2> 316278 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 317197 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:56969
   [junit4]   2> 318217 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 318218 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 318250 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_47204_datanode____.fib4g6/webapp
   [junit4]   2> 318893 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 319939 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:47204
   [junit4]   2> 320108 WARN  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 320109 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 320139 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_34144_datanode____3eslz/webapp
   [junit4]   2> 320571 INFO  (IPC Server handler 5 on 47647) [    ] BlockStateChange BLOCK* processReport: from storage DS-ecfbecd7-f9d7-4c72-8d40-971c8a07a0ed node DatanodeRegistration(127.0.0.1, datanodeUuid=03c9596f-8af8-40e0-8142-d3e21e90ecfc, infoPort=47204, ipcPort=36028, storageInfo=lv=-56;cid=testClusterID;nsid=1853934917;c=0), blocks: 0, hasStaleStorages: true, processing time: 36 msecs
   [junit4]   2> 320571 INFO  (IPC Server handler 5 on 47647) [    ] BlockStateChange BLOCK* processReport: from storage DS-ea307458-e930-4b10-893a-ec5cdc1b5883 node DatanodeRegistration(127.0.0.1, datanodeUuid=03c9596f-8af8-40e0-8142-d3e21e90ecfc, infoPort=47204, ipcPort=36028, storageInfo=lv=-56;cid=testClusterID;nsid=1853934917;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 320981 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 321804 INFO  (SUITE-HdfsCollectionsAPIDistributedZkTest-seed#[E702D057C9439CB2]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:34144
   [junit4]   2> 322294 INFO  (IPC Server handler 7 on 47647) [    ] BlockStateChange BLOCK* processReport: from storage DS-0b645fd5-3acf-461d-8b62-a4e66dbc41c0 node DatanodeRegistration(127.0.0.1, datanodeUuid=0d55d3a0-b413-4c95-b471-558167068f08, infoPort=34144, ipcPort=57662, storageInfo=lv=-56;cid=testClusterID;nsid=1853934917;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs
   [junit4]   2> 322295 INFO  (IPC Server handler 7 on 47647) [    ] BlockStateChange BLOCK* processReport: from storage DS-28347393-7ef8-42fe-911e-2a5bed453e57 node DatanodeRegistration(127.0.0.1, datanodeUuid=0d55d3a0-b413-4c95-b471-558167068f08, infoPort=34144, ipcPort=57662, storageInfo=lv=-56;cid=testClusterID;nsid=1853934917;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 322488 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 322498 INFO  (Thread-850) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 322498 INFO  (Thread-850) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 322675 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.ZkTestServer start zk server on port:39653
   [junit4]   2> 322675 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 322675 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 322684 INFO  (zkCallback-445-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@39ff2a66 name:ZooKeeperConnection Watcher:127.0.0.1:39653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 322684 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 322685 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 322685 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 322700 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 322709 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 322721 INFO  (zkCallback-446-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@477c801a name:ZooKeeperConnection Watcher:127.0.0.1:39653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 322721 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 322721 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 322721 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 322728 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 322729 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 322731 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 322732 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 322733 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 322742 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 322742 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 322743 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 322743 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 322744 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 322744 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 322745 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 322745 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 322746 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 322746 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 322751 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 322751 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 322752 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 322752 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 322754 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 322754 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 322755 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 322755 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 322756 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 322756 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 322771 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 322790 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 322800 INFO  (zkCallback-447-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@431b9a60 name:ZooKeeperConnection Watcher:127.0.0.1:39653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 322800 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 322800 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 322800 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 322803 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 322804 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 322886 INFO  (zkCallback-448-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@6251b172 name:ZooKeeperConnection Watcher:127.0.0.1:39653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 322887 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 322887 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 322887 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf2/solrconfig.xml
   [junit4]   2> 322887 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.xml
   [junit4]   2> 322890 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf2/schema.xml
   [junit4]   2> 322890 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/schema.xml
   [junit4]   2> 322899 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf2/enumsConfig.xml
   [junit4]   2> 322899 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/enumsConfig.xml
   [junit4]   2> 322901 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 322901 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 322901 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf2/stopwords.txt
   [junit4]   2> 322902 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/stopwords.txt
   [junit4]   2> 322902 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf2/protwords.txt
   [junit4]   2> 322903 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/protwords.txt
   [junit4]   2> 322903 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf2/currency.xml
   [junit4]   2> 322903 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/currency.xml
   [junit4]   2> 322910 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf2/open-exchange-rates.json
   [junit4]   2> 322910 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/open-exchange-rates.json
   [junit4]   2> 322918 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 322918 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/mapping-ISOLatin1Accent.txt
   [junit4]   2> 322919 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf2/old_synonyms.txt
   [junit4]   2> 322919 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/old_synonyms.txt
   [junit4]   2> 322920 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf2/synonyms.txt
   [junit4]   2> 322920 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/synonyms.txt
   [junit4]   2> 322921 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/elevate.xml to /configs/conf2/elevate.xml
   [junit4]   2> 322921 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf2/elevate.xml
   [junit4]   2> 323346 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores/collection1
   [junit4]   2> 323348 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 323451 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3c4ae8a9{/,null,AVAILABLE}
   [junit4]   2> 323451 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.e.j.s.ServerConnector Started ServerConnector@7b6e9b00{HTTP/1.1,[http/1.1]}{127.0.0.1:51729}
   [junit4]   2> 323451 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.e.j.s.Server Started @329737ms
   [junit4]   2> 323451 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=hdfs://localhost:47647/hdfs__localhost_47647__x1_jenkins_jenkins-slave_workspace_Lucene-Solr-NightlyTests-6.x_solr_build_solr-core_test_J0_temp_solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001_tempDir-002_control_data, hostContext=/, hostPort=51729, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores}
   [junit4]   2> 323452 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 323452 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001'
   [junit4]   2> 323452 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 323452 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 323452 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 323475 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 323613 INFO  (zkCallback-449-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28994c73 name:ZooKeeperConnection Watcher:127.0.0.1:39653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323613 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 323613 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 323817 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 323817 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/solr.xml
   [junit4]   2> 323821 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores
   [junit4]   2> 323821 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.CoreContainer New CoreContainer 781384845
   [junit4]   2> 323821 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001]
   [junit4]   2> 323821 WARN  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/lib
   [junit4]   2> 323822 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 323822 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 323823 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 323823 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 323823 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39653/solr
   [junit4]   2> 323823 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 323823 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 323823 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 323861 INFO  (zkCallback-452-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@69ad14d6 name:ZooKeeperConnection Watcher:127.0.0.1:39653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323862 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 323862 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 323883 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 323938 INFO  (zkCallback-453-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1514ce54 name:ZooKeeperConnection Watcher:127.0.0.1:39653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 323938 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 323954 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 324002 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 324069 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 324072 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 324075 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 324078 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 324186 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 324191 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 324219 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 324219 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 324220 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 324226 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 324227 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 324228 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 324238 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95678430905630726-127.0.0.1:51729_-n_0000000000
   [junit4]   2> 324238 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:51729_
   [junit4]   2> 324238 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 324239 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.Overseer Overseer (id=95678430905630726-127.0.0.1:51729_-n_0000000000) starting
   [junit4]   2> 324240 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 324251 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 324260 INFO  (OverseerStateUpdate-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 324264 INFO  (OverseerCollectionConfigSetProcessor-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 324305 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51729_
   [junit4]   2> 324306 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:51729_
   [junit4]   2> 324306 INFO  (OverseerStateUpdate-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 324314 INFO  (zkCallback-453-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 324323 INFO  (zkCallback-453-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 324362 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 324362 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 324363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores
   [junit4]   2> 324363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 324363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores/collection1
   [junit4]   2> 324363 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 324505 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 324505 INFO  (TEST-HdfsCollectionsAPIDistributedZkTest.test-seed#[E702D057C9439CB2]) [n:127.0.0.1:51729_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 324506 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 324506 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 324522 INFO  (OverseerStateUpdate-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:51729",
   [junit4]   2>   "node_name":"127.0.0.1:51729_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 324522 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 324522 INFO  (OverseerStateUpdate-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:51729",
   [junit4]   2>   "node_name":"127.0.0.1:51729_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 324523 INFO  (OverseerStateUpdate-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 324523 INFO  (OverseerStateUpdate-95678430905630726-127.0.0.1:51729_-n_0000000000) [n:127.0.0.1:51729_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 324524 INFO  (zkCallback-453-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 325526 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 325526 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 325526 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 325526 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 325527 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 325527 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores/collection1'
   [junit4]   2> 325527 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 325527 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 325528 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 325539 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 325541 WARN  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 325546 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 325566 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 325592 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 325957 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 325961 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 325962 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 326000 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 326007 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 326015 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 326016 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 326027 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 326027 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 326027 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 326428 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:47647/solr_hdfs_home
   [junit4]   2> 326428 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 326429 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest_E702D057C9439CB2-001/control-001/cores/collection1], dataDir=[hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 326429 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@30ecadf6
   [junit4]   2> 326429 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 326725 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 326743 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 326834 WARN  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:47647/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 326834 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 326918 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 326918 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=5, maxMergeAtOnceExplicit=9, maxMergedSegmentMB=2.115234375, floorSegmentMB=1.4345703125, forceMergeDeletesPctAllowed=28.590460133477944, segmentsPerTier=48.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.715408761856564
   [junit4]   2> 327175 INFO  (IPC Server handler 8 on 47647) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:57841 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-ecfbecd7-f9d7-4c72-8d40-971c8a07a0ed:NORMAL:127.0.0.1:41924|RBW], ReplicaUnderConstruction[[DISK]DS-0b645fd5-3acf-461d-8b62-a4e66dbc41c0:NORMAL:127.0.0.1:57841|FINALIZED]]} size 0
   [junit4]   2> 327230 INFO  (IPC Server handler 9 on 47647) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:41924 is added to blk_1073741825_1001 size 71
   [junit4]   2> 327262 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f83b6d3; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 327262 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 327328 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "nodistrib"
   [junit4]   2> 327328 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "dedupe"
   [junit4]   2> 327328 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 327328 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "stored_sig"
   [junit4]   2> 327328 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 327328 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-explicit"
   [junit4]   2> 327329 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain creating updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 327329 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.p.UpdateRequestProcessorChain inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 327329 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore no updateRequestProcessorChain defined as default, creating implicit default
   [junit4]   2> 327329 WARN  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,args = {defaults={a=A,b=B}}}
   [junit4]   2> 327334 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 327335 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 327335 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 327336 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.l.XMLLoader xsltCacheLifetimeSeconds=60
   [junit4]   2> 327342 INFO  (OldIndexDirectoryCleanupThreadForCore-collection1) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Looking for old index directories to cleanup for core collection1 in hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/
   [junit4]   2> 327358 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.RequestHandlers Registered paths: /admin/mbeans,standard,/dump,/update/csv,/sql,/update/json/docs,/admin/luke,/export,/admin/segments,/get,/admin/system,/replication,/admin/properties,/config,/stream,/schema,/admin/plugins,/admin/logging,/update/json,/admin/ping,/admin/threads,/update,/admin/file
   [junit4]   2> 327359 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore Using default statsCache cache: org.apache.solr.search.stats.LocalStatsCache
   [junit4]   2> 327370 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.HdfsUpdateLog
   [junit4]   2> 327370 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 327370 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.HdfsUpdateLog Initializing HdfsUpdateLog: tlogDfsReplication=2
   [junit4]   2> 327412 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 327413 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 327416 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=45, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.0]
   [junit4]   2> 327776 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(HdfsDirectory@hdfs://localhost:47647/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@1f83b6d3; maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 327776 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy newest commit generation = 1
   [junit4]   2> 327776 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@12c347d0[collection1] main]
   [junit4]   2> 327776 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Setting up ZooKeeper-based storage for the RestManager with znodeBase: /configs/conf1
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing RestManager with initArgs: {}
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Reading _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage No data found for znode /configs/conf1/_rest_managed.json
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 327777 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.r.RestManager Initializing 0 registered ManagedResources
   [junit4]   2> 327779 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 327780 INFO  (searcherExecutor-737-thread-1-processing-n:127.0.0.1:51729_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@12c347d0[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 327780 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Looking up max value of version field to seed version buckets
   [junit4]   2> 327781 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo Refreshing highest value of _version_ for 65536 version buckets from index
   [junit4]   2> 327781 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.VersionInfo No terms found for _version_, cannot seed version bucket highest value from index
   [junit4]   2> 327781 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1530854900021329920
   [junit4]   2> 327782 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.UpdateLog Took 1.0ms to seed version buckets with highest version 1530854900021329920
   [junit4]   2> 327782 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController watch zkdir /configs/conf1
   [junit4]   2> 327783 INFO  (coreLoadExecutor-736-thread-1-processing-n:127.0.0.1:51729_) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CoreContainer registering core: collection1
   [junit4]   2> 327783 INFO  (coreZkRegister-730-thread-1-processing-n:127.0.0.1:51729_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController Register replica - core:collection1 address:http://127.0.0.1:51729 collection:control_collection shard:shard1
   [junit4]   2> 327784 INFO  (coreZkRegister-730-thread-1-processing-n:127.0.0.1:51729_ x:collection1 s:shard1 c:control_collection r:core_node1) [n:127.0.0.1:51729_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 327786 INFO  (coreZkRegister-730-thread-1-processing-n:127.0.0.1:51729_ x:collectio

[...truncated too long message...]

nt(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 5326750 INFO  (zkCallback-2325-thread-5-processing-n:127.0.0.1:37077_solr) [n:127.0.0.1:37077_solr    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 5326750 WARN  (zkCallback-2325-thread-5-processing-n:127.0.0.1:37077_solr) [n:127.0.0.1:37077_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 5326746 ERROR (jetty-launcher-2305-thread-5-EventThread) [n:127.0.0.1:57239_solr    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$7/901494263@3170bd78 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@2dc3d4e8[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 5326746 INFO  (jetty-launcher-2305-thread-9) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@597cc3ed{/solr,null,UNAVAILABLE}
   [junit4]   2> 5326752 INFO  (jetty-launcher-2305-thread-10) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@68f095d9{/solr,null,UNAVAILABLE}
   [junit4]   2> 5327614 WARN  (zkCallback-2324-thread-4-processing-n:127.0.0.1:60067_solr) [n:127.0.0.1:60067_solr    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /live_nodes]
   [junit4]   2> 5327615 INFO  (jetty-launcher-2305-thread-6) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4fdeefa2{/solr,null,UNAVAILABLE}
   [junit4]   2> 5327618 INFO  (SUITE-TestTolerantUpdateProcessorCloud-seed#[E702D057C9439CB2]-worker) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33205 33205
   [junit4]   2> 5327933 INFO  (Thread-9521) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:33205 33205
   [junit4]   2> 5327933 WARN  (Thread-9521) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	4	/solr/configs/solrCloudCollectionConfig/managed-schema
   [junit4]   2> 	4	/solr/configs/solrCloudCollectionConfig
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/test_col/state.json
   [junit4]   2> 	2	/solr/overseer_elect/election/95678728983543819-127.0.0.1:37077_solr-n_0000000000
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 
   [junit4]   2> 5327934 INFO  (SUITE-TestTolerantUpdateProcessorCloud-seed#[E702D057C9439CB2]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: test params are: codec=FastDecompressionCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=18046, maxDocsPerChunk=1, blockSize=449), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=18046, blockSize=449)), sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=es-VE, timezone=America/Dominica
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=313688736,total=508559360
   [junit4]   2> NOTE: All tests run in this JVM: [TestExactSharedStatsCache, BasicFunctionalityTest, TestHighFrequencyDictionaryFactory, NumericFieldsTest, SpatialFilterTest, TestSchemaNameResource, DistanceFunctionTest, TestDocumentBuilder, TermVectorComponentTest, TestConfigSetsAPIExclusivity, TestSolrQueryResponse, RulesTest, TestRandomCollapseQParserPlugin, TestTrieFacet, ZkSolrClientTest, ExplicitHLLTest, TestBM25SimilarityFactory, BigEndianAscendingWordDeserializerTest, TestSchemaVersionResource, ZkCLITest, ResponseHeaderTest, TestQueryUtils, TestPKIAuthenticationPlugin, CoreAdminRequestStatusTest, TestUpdate, TestLFUCache, DistributedQueryComponentCustomSortTest, TestConfigOverlay, SolrIndexConfigTest, TimeZoneUtilsTest, DistributedMLTComponentTest, HdfsChaosMonkeyNothingIsSafeTest, StressHdfsTest, LeaderFailoverAfterPartitionTest, UUIDUpdateProcessorFallbackTest, TestTrie, ConvertedLegacyTest, TestGeoJSONResponseWriter, SpellPossibilityIteratorTest, TestSolrCoreProperties, TestQueryTypes, TestDistributedGrouping, MigrateRouteKeyTest, SortByFunctionTest, FileUtilsTest, TestTolerantUpdateProcessorRandomCloud, OverseerStatusTest, TestReqParamsAPI, BufferStoreTest, RemoteQueryErrorTest, TestEmbeddedSolrServerConstructors, FastVectorHighlighterTest, CollectionsAPIAsyncDistributedZkTest, DeleteShardTest, OpenExchangeRatesOrgProviderTest, SolrPluginUtilsTest, TestSizeLimitedDistributedMap, TestBlendedInfixSuggestions, TestBinaryResponseWriter, OpenCloseCoreStressTest, TestSolrFieldCacheMBean, SolrCmdDistributorTest, TestWordDelimiterFilterFactory, HighlighterMaxOffsetTest, TestQuerySenderNoQuery, DataDrivenBlockJoinTest, UpdateRequestProcessorFactoryTest, SimplePostToolTest, TestManagedResourceStorage, TestMinMaxOnMultiValuedField, UniqFieldsUpdateProcessorFactoryTest, MultiTermTest, AliasIntegrationTest, TestHashPartitioner, BasicAuthIntegrationTest, HLLUtilTest, TestCloudPivotFacet, TestLeaderElectionZkExpiry, TestFieldSortValues, ZkStateWriterTest, MBeansHandlerTest, FacetPivotSmallTest, TestInitQParser, TermsComponentTest, CursorMarkTest, TestUseDocValuesAsStored2, TestAuthenticationFramework, IndexBasedSpellCheckerTest, ChaosMonkeySafeLeaderTest, TestConfigSets, TestPartialUpdateDeduplication, SoftAutoCommitTest, SyncSliceTest, OverseerTest, ShardRoutingTest, TestReplicationHandler, CustomCollectionTest, DeleteStatusTest, DistribJoinFromCollectionTest, DistributedQueueTest, ForceLeaderTest, LeaderInitiatedRecoveryOnCommitTest, OutOfBoxZkACLAndCredentialsProvidersTest, OverriddenZkACLAndCredentialsProvidersTest, OverseerTaskQueueTest, RestartWhileUpdatingTest, TestConfigSetsAPIZkFailure, TestDistribDocBasedVersion, TestExclusionRuleCollectionAccess, TestRandomRequestDistribution, TestTolerantUpdateProcessorCloud]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestTolerantUpdateProcessorCloud -Dtests.seed=E702D057C9439CB2 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es-VE -Dtests.timezone=America/Dominica -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J1 | TestTolerantUpdateProcessorCloud (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([E702D057C9439CB2]:0)
   [junit4]    > 	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
   [junit4]    > 	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.createMiniSolrCloudCluster(TestTolerantUpdateProcessorCloud.java:129)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4] Completed [476/585 (5!)] on J1 in 456.96s, 0 tests, 1 failure <<< FAILURES!

[...truncated 359 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2342 tests, 1 suite-level error, 1 error, 3 failures, 31 ignored (16 assumptions) [seed: E702D057C9439CB2]

Total time: 313 minutes 31 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 29 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/29/

6 tests failed.
FAILED:  org.apache.solr.client.solrj.ConnectionReuseTest.test

Error Message:
We expected all communication via streaming client to use one connection! expected=291 got=285

Stack Trace:
java.lang.AssertionError: We expected all communication via streaming client to use one connection! expected=291 got=285
	at __randomizedtesting.SeedInfo.seed([4356054623F48D98:CB023A9C8D08E060]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.client.solrj.ConnectionReuseTest.test(ConnectionReuseTest.java:150)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: https://127.0.0.1:45298/ru_vmt/hg

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: https://127.0.0.1:45298/ru_vmt/hg
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:497)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
	... 47 more


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper]
	at __randomizedtesting.SeedInfo.seed([4356054623F48D98]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNull(Assert.java:551)
	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:248)
	at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores:     1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4356054623F48D98]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)         at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
	at __randomizedtesting.SeedInfo.seed([4356054623F48D98]:0)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([4356054623F48D98:2819A53B5AFB50A2]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11224 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateReaderTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_4356054623F48D98-001/init-core-data-001
   [junit4]   2> 218993 INFO  (SUITE-ZkStateReaderTest-seed#[4356054623F48D98]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true)
   [junit4]   2> 218995 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 218995 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 218996 INFO  (Thread-285) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 218996 INFO  (Thread-285) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 219099 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer start zk server on port:37380
   [junit4]   2> 219099 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 219111 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 219143 INFO  (zkCallback-138-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@23d6d729 name:ZooKeeperConnection Watcher:127.0.0.1:37380 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 219147 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 219147 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 219152 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 219154 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 219159 INFO  (zkCallback-139-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@240d9fa4 name:ZooKeeperConnection Watcher:127.0.0.1:37380 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 219159 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 219159 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 219159 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 219164 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 219165 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 219172 INFO  (zkCallback-140-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@2d37f88f name:ZooKeeperConnection Watcher:127.0.0.1:37380/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 219172 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 219173 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 219173 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 219174 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 219174 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 219175 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 219176 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 219176 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 219176 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 219177 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 219178 INFO  (zkCallback-140-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 219185 INFO  (zkCallback-140-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 219229 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 219229 INFO  (zkCallback-140-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 219231 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37380 37380
   [junit4]   2> 219507 INFO  (Thread-285) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:37380 37380
   [junit4]   2> 219513 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelayLazy
   [junit4]   2> 219514 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithTimeDelay
   [junit4]   2> 219515 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 219523 INFO  (Thread-286) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 219523 INFO  (Thread-286) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 219627 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer start zk server on port:40485
   [junit4]   2> 219627 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 219661 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 219665 INFO  (zkCallback-141-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d3c6e94 name:ZooKeeperConnection Watcher:127.0.0.1:40485 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 219665 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 219665 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 219669 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 219669 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [    ] o.a.z.s.NIOServerCnxn caught end of stream exception
   [junit4]   2> EndOfStreamException: Unable to read additional data from client sessionid 0x153e37aea430000, likely client has closed socket
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:228)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 219670 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 219681 INFO  (zkCallback-142-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28d8853 name:ZooKeeperConnection Watcher:127.0.0.1:40485 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 219681 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 219681 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 219681 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 219684 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 219687 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 219689 INFO  (zkCallback-143-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5e078747 name:ZooKeeperConnection Watcher:127.0.0.1:40485/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 219689 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 219689 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 219690 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 219691 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 219692 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 219693 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 219694 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 219694 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 219695 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 219696 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 219696 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 219696 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 219696 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 219697 INFO  (zkCallback-143-thread-1) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 219699 INFO  (zkCallback-143-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 219699 INFO  (zkCallback-143-thread-1) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 219799 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 219815 INFO  (zkCallback-143-thread-2) [    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 219816 INFO  (zkCallback-143-thread-2) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 219939 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40485 40485
   [junit4]   2> 220148 INFO  (Thread-286) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:40485 40485
   [junit4]   2> 220155 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithTimeDelay-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithTimeDelay
   [junit4]   2> 220156 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testExternalCollectionWatchedNotWatched
   [junit4]   2> 220157 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 220180 INFO  (Thread-287) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 220180 INFO  (Thread-287) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 220280 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer start zk server on port:46622
   [junit4]   2> 220280 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 220303 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 220339 INFO  (zkCallback-144-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@38155407 name:ZooKeeperConnection Watcher:127.0.0.1:46622 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 220339 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 220339 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 220347 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 220361 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 220509 INFO  (zkCallback-145-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5d9a0ddd name:ZooKeeperConnection Watcher:127.0.0.1:46622 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 220509 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 220510 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 220510 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 220513 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 220691 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 220707 INFO  (zkCallback-146-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@e572cad name:ZooKeeperConnection Watcher:127.0.0.1:46622/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 220708 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 220708 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 220712 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 220713 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 220713 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 220714 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 220714 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 220715 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 220716 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 220716 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 220717 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 220717 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 220717 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 220717 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Removing watch for uninteresting collection [c1]
   [junit4]   2> 220736 INFO  (zkCallback-146-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 220736 WARN  (zkCallback-146-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 220738 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46622 46622
   [junit4]   2> 221216 INFO  (Thread-287) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46622 46622
   [junit4]   2> 221217 INFO  (TEST-ZkStateReaderTest.testExternalCollectionWatchedNotWatched-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testExternalCollectionWatchedNotWatched
   [junit4]   2> 221219 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> 221219 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 221235 INFO  (Thread-288) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 221235 INFO  (Thread-288) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 221331 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer start zk server on port:39541
   [junit4]   2> 221332 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 221347 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 221379 INFO  (zkCallback-147-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7b56f3ca name:ZooKeeperConnection Watcher:127.0.0.1:39541 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 221379 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 221379 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 221689 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 221706 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 221720 INFO  (zkCallback-148-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@28b7146b name:ZooKeeperConnection Watcher:127.0.0.1:39541 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 221720 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 221720 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 221720 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 221724 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 221735 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 221752 INFO  (zkCallback-149-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@19c5d055 name:ZooKeeperConnection Watcher:127.0.0.1:39541/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 221752 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 221752 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 221753 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 221753 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 221754 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 221754 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 221755 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 221755 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 221756 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 221756 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader addZkWatch [c1]
   [junit4]   2> 221756 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 221756 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Deleting data for [c1]
   [junit4]   2> 221756 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 221758 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 221759 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Add data for [c1] ver [0]
   [junit4]   2> 221769 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1914119445@7016643f rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@3514e459[Shutting down, pool size = 2, active threads = 1, queued tasks = 0, completed tasks = 1]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 221774 INFO  (zkCallback-149-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 221774 WARN  (zkCallback-149-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 221774 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39541 39541
   [junit4]   2> 221944 INFO  (Thread-288) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:39541 39541
   [junit4]   2> 221948 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefresh-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefresh
   [junit4]   2> 221951 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Starting testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> 221952 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 221958 INFO  (Thread-289) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 221958 INFO  (Thread-289) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 222058 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer start zk server on port:59121
   [junit4]   2> 222058 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 222065 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 222068 INFO  (zkCallback-150-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7790e778 name:ZooKeeperConnection Watcher:127.0.0.1:59121 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 222068 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 222069 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 222071 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 222084 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 222103 INFO  (zkCallback-151-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@5dd6c9c3 name:ZooKeeperConnection Watcher:127.0.0.1:59121 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 222103 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 222103 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 222103 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 222106 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 222120 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 222147 INFO  (zkCallback-152-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@3d761f9 name:ZooKeeperConnection Watcher:127.0.0.1:59121/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 222147 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 222147 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 222148 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 222149 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections
   [junit4]   2> 222150 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 222150 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 222151 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 222152 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 222152 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 222153 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/c1
   [junit4]   2> 222155 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.o.ZkStateWriter going to create_collection /collections/c1/state.json
   [junit4]   2> 222171 ERROR (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]-EventThread) [    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1914119445@5be95595 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@1eebc08c[Shutting down, pool size = 2, active threads = 2, queued tasks = 0, completed tasks = 0]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 222188 INFO  (zkCallback-152-thread-2) [    ] o.a.s.c.c.ZkStateReader A collections change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/collections], has occurred - updating...
   [junit4]   2> 222188 WARN  (zkCallback-152-thread-2) [    ] o.a.s.c.c.ZkStateReader ZooKeeper watch triggered, but Solr cannot talk to ZK: [KeeperErrorCode = Session expired for /collections]
   [junit4]   2> 222215 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59121 59121
   [junit4]   2> 222507 INFO  (Thread-289) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:59121 59121
   [junit4]   2> 222508 INFO  (TEST-ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 ###Ending testStateFormatUpdateWithExplicitRefreshLazy
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ZkStateReaderTest -Dtests.method=testStateFormatUpdateWithExplicitRefreshLazy -Dtests.seed=4356054623F48D98 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=es-ES -Dtests.timezone=America/Manaus -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.57s J0 | ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy <<<
   [junit4]    > Throwable #1: org.apache.solr.common.SolrException: Could not find collection : c1
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4356054623F48D98:2819A53B5AFB50A2]:0)
   [junit4]    > 	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
   [junit4]    > 	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 222516 INFO  (SUITE-ZkStateReaderTest-seed#[4356054623F48D98]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.cloud.overseer.ZkStateReaderTest_4356054623F48D98-001
   [junit4]   2> Apr 04, 2016 10:53:19 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60), sim=ClassicSimilarity, locale=es-ES, timezone=America/Manaus
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=130035168,total=321912832
   [junit4]   2> NOTE: All tests run in this JVM: [DirectoryFactoryTest, RAMDirectoryFactoryTest, RankQueryTest, TestFoldingMultitermQuery, AddSchemaFieldsUpdateProcessorFactoryTest, TestAnalyzedSuggestions, UUIDFieldTest, EchoParamsTest, FileBasedSpellCheckerTest, LeaderInitiatedRecoveryOnShardRestartTest, TestTestInjection, DeleteReplicaTest, TolerantUpdateProcessorTest, URLClassifyProcessorTest, TestMergePolicyConfig, ZkStateReaderTest]
   [junit4] Completed [31/585 (1!)] on J0 in 4.18s, 5 tests, 1 error <<< FAILURES!

[...truncated 32 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.ConnectionReuseTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/init-core-data-001
   [junit4]   2> 256610 INFO  (SUITE-ConnectionReuseTest-seed#[4356054623F48D98]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 256627 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 256633 INFO  (Thread-335) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 256633 INFO  (Thread-335) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 256735 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkTestServer start zk server on port:41367
   [junit4]   2> 256735 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 256785 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 256810 INFO  (zkCallback-169-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@c8060e1 name:ZooKeeperConnection Watcher:127.0.0.1:41367 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 256812 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 256812 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 256813 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 256815 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 256831 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 256843 INFO  (zkCallback-170-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1b7c127b name:ZooKeeperConnection Watcher:127.0.0.1:41367/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 256843 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 256843 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 256843 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 256845 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 256845 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 256846 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 256846 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 256846 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 256853 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 256853 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 256856 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 256856 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 256858 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 256858 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 256859 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 256859 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 256860 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 256860 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 256861 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 256861 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 256863 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 256864 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 256864 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 256864 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 256865 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 256865 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 256866 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 256866 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 257322 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/cores/collection1
   [junit4]   2> 257344 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 257378 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@203a5459{/,null,AVAILABLE}
   [junit4]   2> 257379 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.e.j.s.ServerConnector Started ServerConnector@7d8891ae{HTTP/1.1,[http/1.1]}{127.0.0.1:49156}
   [junit4]   2> 257379 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.e.j.s.Server Started @263236ms
   [junit4]   2> 257380 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/tempDir-001/control/data, hostContext=/, hostPort=49156, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/cores}
   [junit4]   2> 257380 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 257384 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001'
   [junit4]   2> 257384 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 257384 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 257384 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 257402 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 257427 INFO  (zkCallback-171-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@52dc5e7a name:ZooKeeperConnection Watcher:127.0.0.1:41367/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 257427 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 257427 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 257429 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 257429 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/solr.xml
   [junit4]   2> 257433 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/cores
   [junit4]   2> 257433 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.CoreContainer New CoreContainer 177642203
   [junit4]   2> 257433 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001]
   [junit4]   2> 257433 WARN  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/lib
   [junit4]   2> 257433 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 257437 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 257437 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 257437 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 257437 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:41367/solr
   [junit4]   2> 257437 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 257437 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 257443 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 257466 INFO  (zkCallback-174-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@7ef21b1a name:ZooKeeperConnection Watcher:127.0.0.1:41367 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 257466 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 257466 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 257468 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 257475 INFO  (zkCallback-175-thread-1-processing-n:127.0.0.1:49156_) [n:127.0.0.1:49156_    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@23a550e1 name:ZooKeeperConnection Watcher:127.0.0.1:41367/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 257475 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 257476 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 257477 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 257479 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 257491 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 257492 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 257500 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 257500 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 257501 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 257501 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 257502 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 257502 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 257503 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 257504 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 257504 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 257504 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95670136592596996-127.0.0.1:49156_-n_0000000000
   [junit4]   2> 257505 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:49156_
   [junit4]   2> 257505 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 257513 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.Overseer Overseer (id=95670136592596996-127.0.0.1:49156_-n_0000000000) starting
   [junit4]   2> 257513 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 257621 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=30000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 257623 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49156_
   [junit4]   2> 257623 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:49156_
   [junit4]   2> 257623 INFO  (OverseerCollectionConfigSetProcessor-95670136592596996-127.0.0.1:49156_-n_0000000000) [n:127.0.0.1:49156_    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 257636 INFO  (OverseerStateUpdate-95670136592596996-127.0.0.1:49156_-n_0000000000) [n:127.0.0.1:49156_    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 257638 INFO  (zkCallback-175-thread-1-processing-n:127.0.0.1:49156_) [n:127.0.0.1:49156_    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 257643 INFO  (OverseerStateUpdate-95670136592596996-127.0.0.1:49156_-n_0000000000) [n:127.0.0.1:49156_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 257647 INFO  (zkCallback-175-thread-1-processing-n:127.0.0.1:49156_) [n:127.0.0.1:49156_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 257657 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 257657 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 257658 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/cores
   [junit4]   2> 257658 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 257658 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/../../../../../../../../../../x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0/temp/solr.client.solrj.ConnectionReuseTest_4356054623F48D98-001/control-001/cores/collection1
   [junit4]   2> 257659 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 257663 INFO  (coreLoadExecutor-578-thread-1-processing-n:127.0.0.1:49156_) [n:127.0.0.1:49156_ c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 257663 INFO  (coreLoadExecutor-578-thread-1-processing-n:127.0.0.1:49156_) [n:127.0.0.1:49156_ c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 257679 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J0
   [junit4]   2> 257679 INFO  (TEST-ConnectionReuseTest.test-seed#[4356054623F48D98]) [n:127.0.0.1:49156_    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 257695 INFO  (coreLoadExecutor-578-thread-1-processing-n:127.0.0.1:49156_) [n:127.0.0.1:49156_ c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 257703 INFO  (OverseerStateUpdate-95670136592596996-127.0.0.1:49156_-n_0000000000) [n:127.0.0.1:49156_    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:49156",
   [junit4]   2>   "node_name":"127.0.0.1:49156_",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 257711 INFO  (OverseerStateUpdate-95670136592596996-127.0.0.1:49156_-n_0000000000) [n:127.0.0.1:49156_    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4

[...truncated too long message...]

ue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> Apr 04, 2016 11:47:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]   2> Apr 04, 2016 11:47:05 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]   2>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {v_t=Lucene50(blocksize=128), id=Lucene50(blocksize=128), v_s=PostingsFormat(name=Memory doPackFST= false), v_f=TestBloomFilteredLucenePostings(BloomFilteringPostingsFormat(Lucene50(blocksize=128)))}, docValues:{}, maxPointsInLeafNode=926, maxMBSortInHeap=6.016991360682125, sim=RandomSimilarity(queryNorm=true,coord=crazy): {}, locale=en-CA, timezone=JST
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=2,free=95614784,total=530579456
   [junit4]   2> NOTE: All tests run in this JVM: [SolrCoreCheckLockOnStartupTest, TestCoreDiscovery, TestOverriddenPrefixQueryForCustomFieldType, TestSmileRequest, RegexBoostProcessorTest, DirectUpdateHandlerTest, TestFilteredDocIdSet, TestGroupingSearch, AutoCommitTest, ReplicationFactorTest, BigEndianAscendingWordSerializerTest, MultiThreadedOCPTest, DateMathParserTest, DistanceUnitsTest, BinaryUpdateRequestHandlerTest, HdfsDirectoryFactoryTest, ConjunctionSolrSpellCheckerTest, BlockJoinFacetDistribTest, TestJmxIntegration, ClusterStateTest, SharedFSAutoReplicaFailoverTest, TestFieldTypeCollectionResource, TestExactStatsCache, BadIndexSchemaTest, ConnectionManagerTest, TestJsonRequest, TestCopyFieldCollectionResource, TestIndexSearcher, ClassificationUpdateProcessorFactoryTest, TestConfigSetImmutable, StatelessScriptUpdateProcessorFactoryTest, TestReload, SuggesterTest, QueryParsingTest, SimpleFacetsTest, TestSortingResponseWriter, DebugComponentTest, SolrCloudExampleTest, TestClassicSimilarityFactory, BadCopyFieldTest, TestPHPSerializedResponseWriter, TestZkChroot, IgnoreCommitOptimizeUpdateProcessorFactoryTest, MergeStrategyTest, SegmentsInfoRequestHandlerTest, TestSearchPerf, TestCollectionAPI, TestSchemaResource, DistributedQueryComponentOptimizationTest, TestWriterPerf, TestUtils, WrapperMergePolicyFactoryTest, CdcrRequestHandlerTest, TestCSVResponseWriter, TestDownShardTolerantSearch, TestStressRecovery, UpdateParamsTest, TestDocBasedVersionConstraints, HdfsRecoverLeaseTest, TestSolrConfigHandlerConcurrent, HdfsBasicDistributedZk2Test, CollectionsAPISolrJTest, TestInitParams, CollectionStateFormat2Test, SuggesterWFSTTest, TestSchemaManager, TestRuleBasedAuthorizationPlugin, TestRebalanceLeaders, TestCloudDeleteByQuery, PrimitiveFieldTypeTest, TestPerFieldSimilarityClassic, LeaderElectionTest, TlogReplayBufferedWhileIndexingTest, TestFuzzyAnalyzedSuggestions, CoreAdminHandlerTest, RequestHandlersTest, SearchHandlerTest, TestLeaderInitiatedRecoveryThread, TestTrackingShardHandlerFactory, TestBulkSchemaConcurrent, TestComplexPhraseQParserPlugin, PKIAuthenticationIntegrationTest, CoreMergeIndexesAdminHandlerTest, TestReversedWildcardFilterFactory, TestExtendedDismaxParser, ScriptEngineTest, StatsComponentTest, SampleTest, TestElisionMultitermQuery, TestSolrJ, TestSerializedLuceneMatchVersion, TestReloadDeadlock, SliceStateTest, AssignTest, LukeRequestHandlerTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OverseerTest, LeaderElectionIntegrationTest, BasicZkTest, RecoveryZkTest, FullSolrCloudDistribCmdsTest, ClusterStateUpdateTest, TestRandomFaceting, TestRandomDVFaceting, ZkCLITest, TestRecovery, TestHashPartitioner, DistributedSpellCheckComponentTest, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, HardAutoCommitTest, TestRangeQuery, TestCoreContainer, QueryElevationComponentTest, ConvertedLegacyTest, TestSort, TestLazyCores]
   [junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestLazyCores -Dtests.seed=4356054623F48D98 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/x1/jenkins/lucene-data/enwiki.random.lines.txt -Dtests.locale=en-CA -Dtests.timezone=JST -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | TestLazyCores (suite) <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [MockDirectoryWrapper, SolrCore, MDCAwareThreadPoolExecutor, MockDirectoryWrapper]
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4356054623F48D98]:0)
   [junit4]    > 	at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:248)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 
   [junit4]    >    1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4356054623F48D98]:0)Throwable #3: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16371, name=searcherExecutor-6041-thread-1, state=WAITING, group=TGRP-TestLazyCores]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   [junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   [junit4]    >         at java.lang.Thread.run(Thread.java:745)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4356054623F48D98]:0)
   [junit4] Completed [287/585 (4!)] on J2 in 152.27s, 8 tests, 1 failure, 2 errors <<< FAILURES!

[...truncated 898 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2360 tests, 3 suite-level errors, 2 errors, 1 failure, 1055 ignored (13 assumptions) [seed: 4356054623F48D98]

Total time: 261 minutes 30 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any




[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 28 - Still Failing

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/28/

9 tests failed.
FAILED:  org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test

Error Message:
No live SolrServers available to handle this request:[https://127.0.0.1:38388/xi/j/collection1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request:[https://127.0.0.1:38388/xi/j/collection1]
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:7CAA8F30241CB474]:0)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:352)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1165)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:935)
	at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:871)
	at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
	at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.queryServer(AbstractFullDistribZkTestBase.java:1397)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.assertPartialResults(CloudExitableDirectoryReaderTest.java:99)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.doTimeoutTests(CloudExitableDirectoryReaderTest.java:71)
	at org.apache.solr.cloud.CloudExitableDirectoryReaderTest.test(CloudExitableDirectoryReaderTest.java:50)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:38388/xi/j/collection1: org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:577)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
	at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
	... 51 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.TestCloudDeleteByQuery

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:173)
	at org.apache.solr.cloud.TestCloudDeleteByQuery.createMiniSolrCloudCluster(TestCloudDeleteByQuery.java:113)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletesViaShard1NonLeaderClient

Error Message:
org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: XYZ!666 expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: XYZ!666 expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:2F2EA7F1221C332B]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1002)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1007)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletes(TestTolerantUpdateProcessorCloud.java:319)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousDeletesViaShard1NonLeaderClient(TestTolerantUpdateProcessorCloud.java:278)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAddsViaShard1NonLeaderClient

Error Message:
org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: XYZ!666 expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: org.apache.solr.client.solrj.impl.CloudSolrClient@3c28b8b5 should find id: XYZ!666 expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:63C4E63748A7CBA]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1002)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertQueryDocIds(TestTolerantUpdateProcessorCloud.java:1007)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAdds(TestTolerantUpdateProcessorCloud.java:450)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.testVariousAddsViaShard1NonLeaderClient(TestTolerantUpdateProcessorCloud.java:389)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testRandomUpdates

Error Message:
org.apache.solr.client.solrj.impl.HttpSolrClient@54379d12 => [type=<DELQ>,id=<foo_i:[27643 TO ....giberish>,msgSubStr=<null>]: Num error ids: [{type=DELID,id=id_37373,message=No registered leader was found after waiting for 4000ms , collection: test_col slice: shard3}, {type=DELQ,id=foo_i:[27643 TO ....giberish,message=No registered leader was found after waiting for 4000ms , collection: test_col slice: shard3}] expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: org.apache.solr.client.solrj.impl.HttpSolrClient@54379d12 => [type=<DELQ>,id=<foo_i:[27643 TO ....giberish>,msgSubStr=<null>]: Num error ids: [{type=DELID,id=id_37373,message=No registered leader was found after waiting for 4000ms , collection: test_col slice: shard3}, {type=DELQ,id=foo_i:[27643 TO ....giberish,message=No registered leader was found after waiting for 4000ms , collection: test_col slice: shard3}] expected:<1> but was:<2>
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:C83F655AFD9D279E]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:128)
	at org.junit.Assert.assertEquals(Assert.java:472)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorCloud.assertUpdateTolerantErrors(TestTolerantUpdateProcessorCloud.java:958)
	at org.apache.solr.cloud.TestTolerantUpdateProcessorRandomCloud.testRandomUpdates(TestTolerantUpdateProcessorRandomCloud.java:263)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
There were too many update fails (79 > 60) - we expect it can happen, but shouldn't easily

Stack Trace:
java.lang.AssertionError: There were too many update fails (79 > 60) - we expect it can happen, but shouldn't easily
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:7CAA8F30241CB474]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertFalse(Assert.java:68)
	at org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:216)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy

Error Message:
Could not find collection : c1

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : c1
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:9FB11097F3EF04B6]:0)
	at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:170)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:135)
	at org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithExplicitRefreshLazy(ZkStateReaderTest.java:46)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([F4FEB0EA8AE0D98C:7CAA8F30241CB474]:0)
	at org.junit.Assert.fail(Assert.java:92)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertTrue(Assert.java:54)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.testBasics(SharedFSAutoReplicaFailoverTest.java:226)
	at org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest.test(SharedFSAutoReplicaFailoverTest.java:107)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.hdfs.HdfsCollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:48892/r_bve

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting response from server at: http://127.0.0.1:48892/r_bve
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:588)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:381)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:497)
	at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:169)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
	at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
	at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
	at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
	at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
	at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
	at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:482)
	... 47 more




Build Log:
[...truncated 11334 lines...]
   [junit4] Suite: org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest
   [junit4]   2> Creating dataDir: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/init-core-data-001
   [junit4]   2> 302747 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /nyq/y
   [junit4]   2> 305075 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.a.h.u.NativeCodeLoader Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
   [junit4]   1> Formatting using clusterid: testClusterID
   [junit4]   2> 307693 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.a.h.m.i.MetricsConfig Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties
   [junit4]   2> 308257 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
   [junit4]   2> 308264 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 308426 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 308535 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/hdfs to ./temp/Jetty_localhost_57114_hdfs____.quy9t0/webapp
   [junit4]   2> 308756 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 310287 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:57114
   [junit4]   2> 313087 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 313116 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 313165 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_37532_datanode____lj0m6v/webapp
   [junit4]   2> 313623 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 314750 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:37532
   [junit4]   2> 315208 WARN  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.a.h.h.HttpRequestLog Jetty request log can only be enabled using Log4j
   [junit4]   2> 315213 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log jetty-6.1.26
   [junit4]   2> 315272 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Extract jar:file:/x1/jenkins/.ivy2/cache/org.apache.hadoop/hadoop-hdfs/tests/hadoop-hdfs-2.6.0-tests.jar!/webapps/datanode to ./temp/Jetty_localhost_33991_datanode____.wcfrag/webapp
   [junit4]   2> 315618 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log NO JSP Support for /, did not find org.apache.jasper.servlet.JspServlet
   [junit4]   2> 316573 INFO  (SUITE-SharedFSAutoReplicaFailoverTest-seed#[F4FEB0EA8AE0D98C]-worker) [    ] o.m.log Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:33991
   [junit4]   2> 319393 INFO  (IPC Server handler 0 on 49490) [    ] BlockStateChange BLOCK* processReport: from storage DS-f2989bfa-d2be-46d4-801c-10a4533a06b6 node DatanodeRegistration(127.0.0.1, datanodeUuid=eac22cf2-64b4-443d-87e1-906b4fb2595c, infoPort=33991, ipcPort=43835, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), blocks: 0, hasStaleStorages: true, processing time: 19 msecs
   [junit4]   2> 319394 INFO  (IPC Server handler 0 on 49490) [    ] BlockStateChange BLOCK* processReport: from storage DS-1277a29c-bae7-4c57-b662-fc0d797e601b node DatanodeRegistration(127.0.0.1, datanodeUuid=eac22cf2-64b4-443d-87e1-906b4fb2595c, infoPort=33991, ipcPort=43835, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 319394 INFO  (IPC Server handler 9 on 49490) [    ] BlockStateChange BLOCK* processReport: from storage DS-19ad2966-5883-4597-818a-bab8e5c7f705 node DatanodeRegistration(127.0.0.1, datanodeUuid=b18edb30-2702-42ea-bd9e-f04e8cb0f700, infoPort=37532, ipcPort=40472, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), blocks: 0, hasStaleStorages: true, processing time: 10 msecs
   [junit4]   2> 319433 INFO  (IPC Server handler 9 on 49490) [    ] BlockStateChange BLOCK* processReport: from storage DS-d75f0301-8de2-4b9d-8d2e-aaf34945166e node DatanodeRegistration(127.0.0.1, datanodeUuid=b18edb30-2702-42ea-bd9e-f04e8cb0f700, infoPort=37532, ipcPort=40472, storageInfo=lv=-56;cid=testClusterID;nsid=1023663521;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs
   [junit4]   2> 320026 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 320033 INFO  (Thread-735) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 320033 INFO  (Thread-735) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 320132 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.ZkTestServer start zk server on port:32850
   [junit4]   2> 320132 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320144 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320185 INFO  (zkCallback-220-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@4e8789bc name:ZooKeeperConnection Watcher:127.0.0.1:32850 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320186 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320186 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320186 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /solr
   [junit4]   2> 320188 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320202 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320213 INFO  (zkCallback-221-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@52546d32 name:ZooKeeperConnection Watcher:127.0.0.1:32850/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320213 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320213 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320213 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1
   [junit4]   2> 320215 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/collection1/shards
   [junit4]   2> 320215 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection
   [junit4]   2> 320216 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /collections/control_collection/shards
   [junit4]   2> 320216 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 320217 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 320218 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 320218 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/schema.xml
   [junit4]   2> 320221 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 320221 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 320222 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 320222 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 320223 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 320223 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/protwords.txt
   [junit4]   2> 320224 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 320224 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/currency.xml
   [junit4]   2> 320224 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 320224 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/enumsConfig.xml
   [junit4]   2> 320225 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 320225 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 320226 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 320226 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 320227 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 320227 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 320227 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.AbstractZkTestCase put /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 320227 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 320445 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1
   [junit4]   2> 320454 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 320461 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@75078734{/nyq/y,null,AVAILABLE}
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.e.j.s.ServerConnector Started ServerConnector@2ccd46b7{HTTP/1.1,[http/1.1]}{127.0.0.1:34687}
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.e.j.s.Server Started @326947ms
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/nyq/y, hostPort=34687, coreRootDirectory=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores}
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init(): sun.misc.Launcher$AppClassLoader@73d16e93
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001'
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 320462 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320463 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320473 INFO  (zkCallback-222-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@f9918ea name:ZooKeeperConnection Watcher:127.0.0.1:32850/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320473 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320473 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320475 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 320475 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/solr.xml
   [junit4]   2> 320479 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.CorePropertiesLocator Config-defined core root directory: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores
   [junit4]   2> 320479 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.CoreContainer New CoreContainer 2059644463
   [junit4]   2> 320479 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.CoreContainer Loading cores into CoreContainer [instanceDir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001]
   [junit4]   2> 320479 WARN  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.CoreContainer Couldn't add files from /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/lib to classpath: /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/lib
   [junit4]   2> 320479 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.h.c.HttpShardHandlerFactory created with socketTimeout : 90000,urlScheme : ,connTimeout : 15000,maxConnectionsPerHost : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize : 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy : false,useRetries : false,
   [junit4]   2> 320480 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 320480 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.l.LogWatcher SLF4J impl is org.slf4j.impl.Log4jLoggerFactory
   [junit4]   2> 320480 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.l.LogWatcher Registering Log Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
   [junit4]   2> 320481 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:32850/solr
   [junit4]   2> 320481 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.ZkController zkHost includes chroot
   [junit4]   2> 320481 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 320486 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320488 INFO  (zkCallback-225-thread-1) [    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@1f5ba90f name:ZooKeeperConnection Watcher:127.0.0.1:32850 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320489 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320489 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [    ] o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 320555 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 320589 INFO  (zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ConnectionManager Watcher org.apache.solr.common.cloud.ConnectionManager@904a049 name:ZooKeeperConnection Watcher:127.0.0.1:32850/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 320589 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 320597 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue
   [junit4]   2> 320640 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-queue-work
   [junit4]   2> 320705 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-running
   [junit4]   2> 320765 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-completed
   [junit4]   2> 320797 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/collection-map-failure
   [junit4]   2> 320833 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes
   [junit4]   2> 320835 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /aliases.json
   [junit4]   2> 320835 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /clusterstate.json
   [junit4]   2> 320836 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /security.json
   [junit4]   2> 320836 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updating cluster state from ZooKeeper... 
   [junit4]   2> 320849 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (0)
   [junit4]   2> 320877 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect
   [junit4]   2> 320878 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/election
   [junit4]   2> 320879 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 320880 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.LeaderElector Joined leadership election with path: /overseer_elect/election/95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000
   [junit4]   2> 320880 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34687_nyq%2Fy
   [junit4]   2> 320880 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer_elect/leader
   [junit4]   2> 320881 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer Overseer (id=95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) starting
   [junit4]   2> 320883 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /overseer/queue-work
   [junit4]   2> 320886 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.OverseerAutoReplicaFailoverThread Starting OverseerAutoReplicaFailoverThread autoReplicaFailoverWorkLoopDelay=10000 autoReplicaFailoverWaitAfterExpiration=10000 autoReplicaFailoverBadNodeExpiration=60000
   [junit4]   2> 320887 INFO  (OverseerCollectionConfigSetProcessor-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.OverseerTaskProcessor Process current queue of overseer operations
   [junit4]   2> 320887 INFO  (OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer Starting to work on the main queue
   [junit4]   2> 320887 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:34687_nyq%2Fy
   [junit4]   2> 320889 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.SolrZkClient makePath: /live_nodes/127.0.0.1:34687_nyq%2Fy
   [junit4]   2> 320895 INFO  (zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader A live node change: [WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes], has occurred - updating... (live nodes size: [0])
   [junit4]   2> 320901 INFO  (OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 320901 INFO  (zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (1)
   [junit4]   2> 320909 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CoreContainer Security conf doesn't exist. Skipping setup for authorization module.
   [junit4]   2> 320909 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CoreContainer No authentication plugin used.
   [junit4]   2> 320913 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CorePropertiesLocator Looking for core definitions underneath /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores
   [junit4]   2> 320924 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CoreDescriptor Created CoreDescriptor: {name=collection1, config=solrconfig.xml, transient=false, schema=schema.xml, loadOnStartup=true, configSetProperties=configsetprops.json, collection=control_collection, coreNodeName=, dataDir=data/, shard=}
   [junit4]   2> 320924 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CorePropertiesLocator Found core collection1 in /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1
   [junit4]   2> 320924 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions
   [junit4]   2> 320925 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ZkController publishing state=down
   [junit4]   2> 320925 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ZkController numShards not found on descriptor - reading it from system property
   [junit4]   2> 320928 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ZkController look for our core node name
   [junit4]   2> 320936 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.s.SolrDispatchFilter user.dir=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 320936 INFO  (TEST-SharedFSAutoReplicaFailoverTest.test-seed#[F4FEB0EA8AE0D98C]) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.s.SolrDispatchFilter SolrDispatchFilter.init() done
   [junit4]   2> 320937 INFO  (OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.Overseer processMessage: queueSize: 1, message = {
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:34687/nyq/y",
   [junit4]   2>   "node_name":"127.0.0.1:34687_nyq%2Fy",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"} current state version: 0
   [junit4]   2> 320937 INFO  (OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
   [junit4]   2>   "core":"collection1",
   [junit4]   2>   "roles":null,
   [junit4]   2>   "base_url":"http://127.0.0.1:34687/nyq/y",
   [junit4]   2>   "node_name":"127.0.0.1:34687_nyq%2Fy",
   [junit4]   2>   "numShards":"1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "shard":null,
   [junit4]   2>   "collection":"control_collection",
   [junit4]   2>   "operation":"state"}
   [junit4]   2> 320938 INFO  (OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.o.ClusterStateMutator building a new cName: control_collection
   [junit4]   2> 320938 INFO  (OverseerStateUpdate-95660805454299140-127.0.0.1:34687_nyq%2Fy-n_0000000000) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 320939 INFO  (zkCallback-226-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json], has occurred - updating... (live nodes size: [1])
   [junit4]   2> 321929 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ZkController waiting to find shard id in clusterstate for collection1
   [junit4]   2> 321929 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ZkController Check for collection zkNode:control_collection
   [junit4]   2> 321929 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ZkController Collection zkNode exists
   [junit4]   2> 321929 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader Load collection config from: [/collections/control_collection]
   [junit4]   2> 321930 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.c.ZkStateReader path=[/collections/control_collection] [configName]=[conf1] specified config exists in ZooKeeper
   [junit4]   2> 321930 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader new SolrResourceLoader for directory: '/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1'
   [junit4]   2> 321930 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 321930 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.SolrResourceLoader solr home defaulted to 'solr/' (could not find system property or JNDI)
   [junit4]   2> 321931 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.Config loaded config solrconfig.xml with version 0 
   [junit4]   2> 321982 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.SolrConfig current version of requestparams : -1
   [junit4]   2> 321993 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.1.0
   [junit4]   2> 322012 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.SolrConfig Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 322037 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 322576 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 322596 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.IndexSchema default search field in schema is text
   [junit4]   2> 322611 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.IndexSchema unique key field: id
   [junit4]   2> 322633 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 322635 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.FileExchangeRateProvider Reloading exchange rates from file currency.xml
   [junit4]   2> 322637 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 322637 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.s.OpenExchangeRatesOrgProvider Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 322646 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.ConfigSetProperties Did not find ConfigSet properties, assuming default properties: Can't find resource 'configsetprops.json' in classpath or '/configs/conf1', cwd=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2
   [junit4]   2> 322646 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 322646 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore org.apache.solr.core.HdfsDirectoryFactory
   [junit4]   2> 322677 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory solr.hdfs.home=hdfs://localhost:49490/solr_hdfs_home
   [junit4]   2> 322677 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Solr Kerberos Authentication disabled
   [junit4]   2> 322685 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build/solr-core/test/J2/temp/solr.cloud.SharedFSAutoReplicaFailoverTest_F4FEB0EA8AE0D98C-001/control-001/cores/collection1], dataDir=[hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/]
   [junit4]   2> 322685 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@60576839
   [junit4]   2> 322686 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 322756 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data
   [junit4]   2> 322760 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore New index directory detected: old=null new=hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index/
   [junit4]   2> 322783 WARN  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrCore [collection1] Solr index directory 'hdfs:/localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 322783 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory creating directory factory for path hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 322789 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Number of slabs of block cache [1] with direct memory allocation set to [true]
   [junit4]   2> 322789 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Block cache target memory usage, slab size of [8388608] will allocate [1] slabs and use ~[8388608] bytes
   [junit4]   2> 322789 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.HdfsDirectoryFactory Creating new single instance HDFS BlockCache
   [junit4]   2> 326084 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.s.b.BlockDirectory Block cache on write is disabled
   [junit4]   2> 326084 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.CachingDirectoryFactory return new directory for hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index
   [junit4]   2> 326084 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=9, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=1.0]
   [junit4]   2> 326698 INFO  (IPC Server handler 5 on 49490) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:35579 is added to blk_1073741825_1001{blockUCState=COMMITTED, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-d75f0301-8de2-4b9d-8d2e-aaf34945166e:NORMAL:127.0.0.1:49587|RBW], ReplicaUnderConstruction[[DISK]DS-1277a29c-bae7-4c57-b662-fc0d797e601b:NORMAL:127.0.0.1:35579|RBW]]} size 71
   [junit4]   2> 326702 INFO  (IPC Server handler 1 on 49490) [    ] BlockStateChange BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:49587 is added to blk_1073741825_1001 size 71
   [junit4]   2> 327134 INFO  (coreLoadExecutor-834-thread-1-processing-n:127.0.0.1:34687_nyq%2Fy) [n:127.0.0.1:34687_nyq%2Fy c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> 	commit{dir=NRTCachingDirectory(BlockDirectory(HdfsDirectory@hdfs://localhost:49490/solr_hdfs_home/control_collection/core_node1/data/index lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@54ccbd95); maxCacheMB=192.0 maxMergeSizeMB=16.0),segFN=segments_1,generation=1}
   [junit4]   2> 327134 INFO  (c

[...truncated too long message...]

at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 6386175 ERROR (zkCallback-2537-thread-10-processing-n:127.0.0.1:44804_r_bve-EventThread) [n:127.0.0.1:44804_r_bve    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1370100541@4c2958b0 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4f433467[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1773]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 6386175 ERROR (zkCallback-2537-thread-10-processing-n:127.0.0.1:44804_r_bve-EventThread) [n:127.0.0.1:44804_r_bve    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1370100541@6c1db556 rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4f433467[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1773]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> 6386175 ERROR (zkCallback-2537-thread-10-processing-n:127.0.0.1:44804_r_bve-EventThread) [n:127.0.0.1:44804_r_bve    ] o.a.z.ClientCnxn Error while calling watcher 
   [junit4]   2> java.util.concurrent.RejectedExecutionException: Task org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$6/1370100541@42e4e99b rejected from org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor@4f433467[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1773]
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
   [junit4]   2> 	at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:214)
   [junit4]   2> 	at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient$3.process(SolrZkClient.java:266)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
   [junit4]   2> 	at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene60): {other_tl1=PostingsFormat(name=LuceneVarGapDocFreqInterval), range_facet_l_dv=PostingsFormat(name=MockRandom), _version_=PostingsFormat(name=LuceneVarGapDocFreqInterval), multiDefault=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=LuceneVarGapDocFreqInterval), a_i1=PostingsFormat(name=Memory doPackFST= false), id=PostingsFormat(name=MockRandom), range_facet_i_dv=PostingsFormat(name=Memory doPackFST= false), text=FST50, range_facet_l=PostingsFormat(name=Memory doPackFST= false), timestamp=PostingsFormat(name=Memory doPackFST= false)}, docValues:{range_facet_l_dv=DocValuesFormat(name=Lucene54), range_facet_i_dv=DocValuesFormat(name=Lucene54), timestamp=DocValuesFormat(name=Lucene54)}, maxPointsInLeafNode=77, maxMBSortInHeap=5.235541453874914, sim=ClassicSimilarity, locale=sk, timezone=Europe/Chisinau
   [junit4]   2> NOTE: Linux 3.13.0-52-generic amd64/Oracle Corporation 1.8.0_74 (64-bit)/cpus=4,threads=1,free=130955528,total=517996544
   [junit4]   2> NOTE: All tests run in this JVM: [TestPHPSerializedResponseWriter, TestReloadDeadlock, TestCollectionAPI, TestFuzzyAnalyzedSuggestions, DeleteLastCustomShardedReplicaTest, TestXIncludeConfig, TestConfigSetImmutable, AssignTest, CleanupOldIndexTest, AnalyticsMergeStrategyTest, HighlighterConfigTest, DistributedQueryComponentOptimizationTest, BigEndianAscendingWordSerializerTest, SearchHandlerTest, CoreMergeIndexesAdminHandlerTest, XmlUpdateRequestHandlerTest, ClusterStateTest, TestJsonRequest, ScriptEngineTest, AutoCommitTest, DirectUpdateHandlerTest, SharedFSAutoReplicaFailoverTest, DirectUpdateHandlerOptimizeTest, TestStressRecovery, TestSolrConfigHandlerConcurrent, TestXmlQParser, ReplicationFactorTest, OverseerCollectionConfigSetProcessorTest, TestSortingResponseWriter, TestClusterStateMutator, DistribJoinFromCollectionTest, TestExtendedDismaxParser, PKIAuthenticationIntegrationTest, TestDownShardTolerantSearch, UnloadDistributedZkTest, FullSolrCloudDistribCmdsTest, SolrTestCaseJ4Test, RequestHandlersTest, ForceLeaderTest, TestWriterPerf, TestBulkSchemaConcurrent, CollectionsAPISolrJTest, ClassificationUpdateProcessorFactoryTest, TestRequestStatusCollectionAPI, FieldAnalysisRequestHandlerTest, OverseerTaskQueueTest, DebugComponentTest, BlockCacheTest, TestSearchPerf, TestFieldTypeCollectionResource, DateMathParserTest, TestSolrCloudWithKerberosAlt, UpdateParamsTest, SolrCloudExampleTest, HdfsDirectoryFactoryTest, SegmentsInfoRequestHandlerTest, SolrCoreCheckLockOnStartupTest, TestOrdValues, TestDocBasedVersionConstraints, TlogReplayBufferedWhileIndexingTest, TestCopyFieldCollectionResource, TestDefaultStatsCache, TestIndexSearcher, TestOverriddenPrefixQueryForCustomFieldType, ConnectionManagerTest, CachingDirectoryFactoryTest, DocValuesMissingTest, StatsComponentTest, TestSchemaResource, BasicDistributedZk2Test, SampleTest, SparseHLLTest, BadCopyFieldTest, TestJmxIntegration, TestRecovery, HdfsDirectoryTest, TestCSVResponseWriter, IgnoreCommitOptimizeUpdateProcessorFactoryTest, TestDistributedStatsComponentCardinality, TestInitParams, TestFilteredDocIdSet, TestCollationField, TestRebalanceLeaders, HdfsSyncSliceTest, TermVectorComponentDistributedTest, TestSort, HardAutoCommitTest, TestFileDictionaryLookup, TestComplexPhraseQParserPlugin, RecoveryZkTest, NoCacheHeaderTest, TestRuleBasedAuthorizationPlugin, OverseerTest, DirectSolrConnectionTest, LukeRequestHandlerTest, CollectionsAPIDistributedZkTest, LeaderElectionTest, ZkSolrClientTest, TestZkChroot, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, TestDistributedSearch, ConfigSetsAPITest, CustomCollectionTest, MultiThreadedOCPTest, RemoteQueryErrorTest, SolrXmlInZkTest, TestAuthenticationFramework, TestCloudDeleteByQuery, TriLevelCompositeIdRoutingTest, HdfsCollectionsAPIDistributedZkTest]
   [junit4] Completed [563/585 (8!)] on J2 in 847.35s, 1 test, 1 error <<< FAILURES!

[...truncated 85 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:747: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:684: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/build.xml:59: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/build.xml:233: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/solr/common-build.xml:529: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1457: The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-6.x/lucene/common-build.xml:1014: There were test failures: 585 suites (10 ignored), 2354 tests, 1 suite-level error, 3 errors, 5 failures, 31 ignored (16 assumptions) [seed: F4FEB0EA8AE0D98C]

Total time: 296 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
No prior successful build to compare, so performing full copy of artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any