You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2017/02/12 02:43:28 UTC

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 691 - Still Failing!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/691/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration

Error Message:
Error CREATEing SolrCore 'testcore': Unable to create core [testcore] Caused by: invalid API spec: apispec/core.SchemaEdit.json

Stack Trace:
org.apache.solr.common.SolrException: Error CREATEing SolrCore 'testcore': Unable to create core [testcore] Caused by: invalid API spec: apispec/core.SchemaEdit.json
	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B:F1F92BEEE52E055E]:0)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:874)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:807)
	at org.apache.solr.cloud.ClusterStateUpdateTest.testCoreRegistration(ClusterStateUpdateTest.java:156)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.solr.common.SolrException: Unable to create core [testcore]
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:919)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:840)
	... 41 more
Caused by: org.apache.solr.common.SolrException: invalid API spec: apispec/core.SchemaEdit.json
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:943)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:807)
	at org.apache.solr.core.CoreContainer.create(CoreContainer.java:904)
	... 42 more
Caused by: java.lang.RuntimeException: invalid API spec: apispec/core.SchemaEdit.json
	at org.apache.solr.common.util.ValidatingJsonMap.parse(ValidatingJsonMap.java:318)
	at org.apache.solr.api.ApiBag.lambda$getSpec$0(ApiBag.java:229)
	at org.apache.solr.api.Api.getSpec(Api.java:64)
	at org.apache.solr.api.ApiBag.register(ApiBag.java:72)
	at org.apache.solr.core.PluginBag.put(PluginBag.java:215)
	at org.apache.solr.core.PluginBag.init(PluginBag.java:272)
	at org.apache.solr.core.RequestHandlers.initHandlersFromConfig(RequestHandlers.java:130)
	at org.apache.solr.core.SolrCore.<init>(SolrCore.java:902)
	... 44 more


FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Could not find collection:collection2

Stack Trace:
java.lang.AssertionError: Could not find collection:collection2
	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B:C726729B32A86693]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.junit.Assert.assertNotNull(Assert.java:526)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:159)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:144)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.testIndexingBatchPerRequestWithHttpSolrClient(FullSolrCloudDistribCmdsTest.java:620)
	at org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test(FullSolrCloudDistribCmdsTest.java:152)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.cloud.LeaderElectionTest.testElection

Error Message:
java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO

Stack Trace:
org.apache.solr.common.SolrException: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B:277AF971B468BC1D]:0)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:170)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:106)
	at org.apache.solr.cloud.LeaderElectionTest$ElectorSetup.<init>(LeaderElectionTest.java:114)
	at org.apache.solr.cloud.LeaderElectionTest$ClientThread.<init>(LeaderElectionTest.java:151)
	at org.apache.solr.cloud.LeaderElectionTest$ClientThread.<init>(LeaderElectionTest.java:138)
	at org.apache.solr.cloud.LeaderElectionTest.testElection(LeaderElectionTest.java:286)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnxnSocketNIO
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1782)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:447)
	at org.apache.zookeeper.ZooKeeper.<init>(ZooKeeper.java:380)
	at org.apache.solr.common.cloud.SolrZooKeeper.<init>(SolrZooKeeper.java:43)
	at org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
	at org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
	at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:145)
	... 45 more
Caused by: java.io.IOException: Too many open files
	at sun.nio.ch.IOUtil.makePipe(Native Method)
	at sun.nio.ch.KQueueSelectorImpl.<init>(KQueueSelectorImpl.java:84)
	at sun.nio.ch.KQueueSelectorProvider.openSelector(KQueueSelectorProvider.java:42)
	at java.nio.channels.Selector.open(Selector.java:227)
	at org.apache.zookeeper.ClientCnxnSocketNIO.<init>(ClientCnxnSocketNIO.java:43)
	at sun.reflect.GeneratedConstructorAccessor44.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at java.lang.Class.newInstance(Class.java:442)
	at org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1779)
	... 51 more


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionTest

Error Message:
26 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest:     1) Thread[id=12301, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    2) Thread[id=12270, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    3) Thread[id=12313, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    4) Thread[id=12273, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)         at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)    5) Thread[id=12310, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    6) Thread[id=12278, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    7) Thread[id=12302, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    8) Thread[id=12294, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)    9) Thread[id=12297, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   10) Thread[id=12309, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   11) Thread[id=12277, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)   12) Thread[id=12305, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   13) Thread[id=12265, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   14) Thread[id=12274, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   15) Thread[id=12289, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   16) Thread[id=12306, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   17) Thread[id=12281, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   18) Thread[id=12290, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   19) Thread[id=12286, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   20) Thread[id=12269, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   21) Thread[id=12285, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   22) Thread[id=12282, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   23) Thread[id=12293, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   24) Thread[id=12266, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   25) Thread[id=12314, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)   26) Thread[id=12298, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 26 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest: 
   1) Thread[id=12301, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   2) Thread[id=12270, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   3) Thread[id=12313, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   4) Thread[id=12273, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
        at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1185)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1110)
   5) Thread[id=12310, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   6) Thread[id=12278, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   7) Thread[id=12302, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   8) Thread[id=12294, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   9) Thread[id=12297, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  10) Thread[id=12309, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  11) Thread[id=12277, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
  12) Thread[id=12305, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  13) Thread[id=12265, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  14) Thread[id=12274, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  15) Thread[id=12289, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  16) Thread[id=12306, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  17) Thread[id=12281, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  18) Thread[id=12290, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  19) Thread[id=12286, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  20) Thread[id=12269, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  21) Thread[id=12285, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  22) Thread[id=12282, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  23) Thread[id=12293, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  24) Thread[id=12266, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  25) Thread[id=12314, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
  26) Thread[id=12298, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderElectionTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=12301, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    2) Thread[id=12305, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    3) Thread[id=12265, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    4) Thread[id=12313, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    5) Thread[id=12289, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    6) Thread[id=12281, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    7) Thread[id=12273, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)    8) Thread[id=12269, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)    9) Thread[id=12285, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)   10) Thread[id=12293, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)   11) Thread[id=12297, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   12) Thread[id=12309, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)   13) Thread[id=12277, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]         at java.lang.Thread.sleep(Native Method)         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=12301, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   2) Thread[id=12305, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   3) Thread[id=12265, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   4) Thread[id=12313, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   5) Thread[id=12289, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   6) Thread[id=12281, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   7) Thread[id=12273, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   8) Thread[id=12269, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   9) Thread[id=12285, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
  10) Thread[id=12293, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
  11) Thread[id=12297, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  12) Thread[id=12309, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
  13) Thread[id=12277, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.lang.Thread.sleep(Native Method)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B]:0)


FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>

Stack Trace:
java.lang.AssertionError: Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>
	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B:C726729B32A86693]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotSame(Assert.java:641)
	at org.junit.Assert.assertSame(Assert.java:580)
	at org.apache.solr.cloud.MultiThreadedOCPTest.testDeduplicationOfSubmittedTasks(MultiThreadedOCPTest.java:250)
	at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11856 lines...]
   [junit4] Suite: org.apache.solr.cloud.MultiThreadedOCPTest
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/init-core-data-001
   [junit4]   2> 1241938 INFO  (SUITE-MultiThreadedOCPTest-seed#[4F724D419C540B6B]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 1241938 INFO  (SUITE-MultiThreadedOCPTest-seed#[4F724D419C540B6B]-worker) [    ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
   [junit4]   2> 1241946 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 1241946 INFO  (Thread-2418) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 1241946 INFO  (Thread-2418) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 1242052 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.ZkTestServer start zk server on port:63108
   [junit4]   2> 1242144 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 1242183 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 1242186 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 1242189 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 1242193 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 1242196 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 1242199 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 1242202 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 1242206 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 1242209 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 1242213 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractZkTestCase put /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 1245427 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/control-001/cores/collection1
   [junit4]   2> 1245431 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1245434 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@9ef3586{/,null,AVAILABLE}
   [junit4]   2> 1245436 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@20c00ebc{HTTP/1.1,[http/1.1]}{127.0.0.1:63111}
   [junit4]   2> 1245436 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.Server Started @1251835ms
   [junit4]   2> 1245436 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/tempDir-001/control/data, hostContext=/, hostPort=63111, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/control-001/cores}
   [junit4]   2> 1245437 ERROR (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1245437 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 1245437 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1245437 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1245437 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-12T01:58:30.859Z
   [junit4]   2> 1245443 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1245443 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/control-001/solr.xml
   [junit4]   2> 1245456 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1245458 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63108/solr
   [junit4]   2> 1245504 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63111_    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:63111_
   [junit4]   2> 1245506 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63111_    ] o.a.s.c.Overseer Overseer (id=97443165463314436-127.0.0.1:63111_-n_0000000000) starting
   [junit4]   2> 1245520 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63111_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63111_
   [junit4]   2> 1245524 INFO  (zkCallback-1547-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1245637 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63111_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/control-001/cores
   [junit4]   2> 1245638 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63111_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1245642 INFO  (OverseerStateUpdate-97443165463314436-127.0.0.1:63111_-n_0000000000) [n:127.0.0.1:63111_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard1
   [junit4]   2> 1246660 WARN  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1246661 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1246681 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1246755 WARN  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1246757 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1246776 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection control_collection
   [junit4]   2> 1246776 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/control-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/control-001/cores/collection1/data/]
   [junit4]   2> 1246777 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59eaad52
   [junit4]   2> 1246779 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=20.5478515625, floorSegmentMB=1.6240234375, forceMergeDeletesPctAllowed=20.205049605539013, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3834342656412677
   [junit4]   2> 1246784 WARN  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1246874 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1246874 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1246876 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1246876 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1246877 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1729712950, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1246878 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@30e889cc[collection1] main]
   [junit4]   2> 1246880 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1246881 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1246881 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1246882 INFO  (coreLoadExecutor-4717-thread-1-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559090652568879104
   [junit4]   2> 1246883 INFO  (searcherExecutor-4718-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@30e889cc[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1246894 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1246894 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1246894 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:63111/collection1/
   [junit4]   2> 1246894 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 1246894 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.SyncStrategy http://127.0.0.1:63111/collection1/ has no replicas
   [junit4]   2> 1246902 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:63111/collection1/ shard1
   [junit4]   2> 1247056 INFO  (coreZkRegister-4710-thread-1-processing-n:127.0.0.1:63111_ x:collection1 c:control_collection) [n:127.0.0.1:63111_ c:control_collection s:shard1 r:core_node1 x:collection1] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 1247207 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1247212 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:63108/solr ready
   [junit4]   2> 1247212 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 1247212 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase Creating collection1 with stateFormat=2
   [junit4]   2> 1247630 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.SolrTestCaseJ4 Writing core.properties file to /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001/cores/collection1
   [junit4]   2> 1247631 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001
   [junit4]   2> 1247632 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 1247634 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7edc9413{/,null,AVAILABLE}
   [junit4]   2> 1247634 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.AbstractConnector Started ServerConnector@533586c9{HTTP/1.1,[http/1.1]}{127.0.0.1:63117}
   [junit4]   2> 1247634 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.e.j.s.Server Started @1254034ms
   [junit4]   2> 1247634 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {solr.data.dir=/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/tempDir-001/jetty1, solrconfig=solrconfig.xml, hostContext=/, hostPort=63117, coreRootDirectory=/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001/cores}
   [junit4]   2> 1247634 ERROR (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 1247634 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 1247635 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 1247635 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 1247635 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-12T01:58:33.057Z
   [junit4]   2> 1247639 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 1247639 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.SolrXmlConfig Loading container configuration from /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001/solr.xml
   [junit4]   2> 1247730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 1247730 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:63108/solr
   [junit4]   2> 1247747 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63117_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 1247756 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63117_    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:63117_
   [junit4]   2> 1247761 INFO  (zkCallback-1547-thread-2-processing-n:127.0.0.1:63111_) [n:127.0.0.1:63111_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1247762 INFO  (zkCallback-1551-thread-1) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1247762 INFO  (zkCallback-1557-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 1247837 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63117_    ] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001/cores
   [junit4]   2> 1247837 INFO  (TEST-MultiThreadedOCPTest.test-seed#[4F724D419C540B6B]) [n:127.0.0.1:63117_    ] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
   [junit4]   2> 1247841 INFO  (OverseerStateUpdate-97443165463314436-127.0.0.1:63111_-n_0000000000) [n:127.0.0.1:63111_    ] o.a.s.c.o.ReplicaMutator Assigning new node to shard shard=shard2
   [junit4]   2> 1247947 INFO  (zkCallback-1557-thread-2-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 1248856 WARN  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.Config Beginning with Solr 5.5, <mergePolicy> is deprecated, use <mergePolicyFactory> instead.
   [junit4]   2> 1248856 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 1248867 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] Schema name=test
   [junit4]   2> 1248934 WARN  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.s.IndexSchema [collection1] default search field in schema is text. WARNING: Deprecated, please use 'df' on request instead.
   [junit4]   2> 1248936 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
   [junit4]   2> 1248955 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from collection collection1
   [junit4]   2> 1248955 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.SolrCore [[collection1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001/cores/collection1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/../../../../../../../../../Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_4F724D419C540B6B-001/shard-1-001/cores/collection1/data/]
   [junit4]   2> 1248956 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.JmxMonitoredMap JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@59eaad52
   [junit4]   2> 1248957 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=50, maxMergeAtOnceExplicit=43, maxMergedSegmentMB=20.5478515625, floorSegmentMB=1.6240234375, forceMergeDeletesPctAllowed=20.205049605539013, segmentsPerTier=11.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3834342656412677
   [junit4]   2> 1248963 WARN  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A,b=B}}}
   [junit4]   2> 1249029 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 1249029 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 1249031 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 1249031 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 1249032 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.AlcoholicMergePolicy: [AlcoholicMergePolicy: minMergeSize=0, mergeFactor=10, maxMergeSize=1729712950, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.1]
   [junit4]   2> 1249033 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.s.SolrIndexSearcher Opening [Searcher@746680f1[collection1] main]
   [junit4]   2> 1249035 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 1249035 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 1249035 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 1249036 INFO  (searcherExecutor-4729-thread-1-processing-n:127.0.0.1:63117_ x:collection1 c:collection1) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.c.SolrCore [collection1] Registered new searcher Searcher@746680f1[collection1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 1249037 INFO  (coreLoadExecutor-4728-thread-1-processing-n:127.0.0.1:63117_) [n:127.0.0.1:63117_ c:collection1   x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559090654828560384
   [junit4]   2> 1249047 INFO  (coreZkRegister-4723-thread-1-processing-n:127.0.0.1:63117_ x:collection1 c:collection1) [n:127.0.0.1:63117_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 1249047 INFO  (coreZkRegister-4723-thread-1-processing-n:127.0.0.1:63117_ x:collection1 c:collection1) [n:127.0.0.1:63117_ c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 1249048 INFO  (

[...truncated too long message...]

D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >   25) Thread[id=12314, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    >   26) Thread[id=12298, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at sun.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]    >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]    >         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=12301, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    2) Thread[id=12305, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    3) Thread[id=12265, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    4) Thread[id=12313, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    5) Thread[id=12289, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    6) Thread[id=12281, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    7) Thread[id=12273, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >    8) Thread[id=12269, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >    9) Thread[id=12285, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >   10) Thread[id=12293, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)
   [junit4]    >   11) Thread[id=12297, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >   12) Thread[id=12309, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    >   13) Thread[id=12277, name=TEST-LeaderElectionTest.testElection-seed#[4F724D419C540B6B]-SendThread(127.0.0.1:64939), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
   [junit4]    >         at java.lang.Thread.sleep(Native Method)
   [junit4]    >         at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
   [junit4]    >         at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([4F724D419C540B6B]:0)
   [junit4] Completed [363/691 (4!)] on J1 in 84.01s, 5 tests, 3 errors <<< FAILURES!

[...truncated 63328 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/docs/solr-solrj/overview-summary.html
     [exec]   missing: org.apache.solr.client.solrj.io.eval
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:775: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:101: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:654: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:670: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:2490: exec returned: 1

Total time: 89 minutes 8 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 693 - Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/693/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.MultiThreadedOCPTest.test

Error Message:
Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>

Stack Trace:
java.lang.AssertionError: Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>
	at __randomizedtesting.SeedInfo.seed([3CF871AE4C014DE1:B4AC4E74E2FD2019]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.failNotSame(Assert.java:641)
	at org.junit.Assert.assertSame(Assert.java:580)
	at org.apache.solr.cloud.MultiThreadedOCPTest.testDeduplicationOfSubmittedTasks(MultiThreadedOCPTest.java:250)
	at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:64)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReqParamsAPI.test

Error Message:
Could not get expected value  'A val' for path 'params/a' full output: {   "responseHeader":{     "status":0,     "QTime":0},   "params":{     "wt":"json",     "useParams":""},   "context":{     "webapp":"/solr",     "path":"/dump0",     "httpMethod":"GET"}},  from server:  http://127.0.0.1:54239/solr/collection1_shard1_replica1

Stack Trace:
java.lang.AssertionError: Could not get expected value  'A val' for path 'params/a' full output: {
  "responseHeader":{
    "status":0,
    "QTime":0},
  "params":{
    "wt":"json",
    "useParams":""},
  "context":{
    "webapp":"/solr",
    "path":"/dump0",
    "httpMethod":"GET"}},  from server:  http://127.0.0.1:54239/solr/collection1_shard1_replica1
	at __randomizedtesting.SeedInfo.seed([3CF871AE4C014DE1:B4AC4E74E2FD2019]:0)
	at org.junit.Assert.fail(Assert.java:93)
	at org.junit.Assert.assertTrue(Assert.java:43)
	at org.apache.solr.core.TestSolrConfigHandler.testForResponseElement(TestSolrConfigHandler.java:556)
	at org.apache.solr.handler.TestReqParamsAPI.testReqParams(TestReqParamsAPI.java:127)
	at org.apache.solr.handler.TestReqParamsAPI.test(TestReqParamsAPI.java:69)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11058 lines...]
   [junit4] Suite: org.apache.solr.handler.TestReqParamsAPI
   [junit4]   2> Creating dataDir: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/init-core-data-001
   [junit4]   2> 290558 INFO  (SUITE-TestReqParamsAPI-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 290559 INFO  (SUITE-TestReqParamsAPI-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001
   [junit4]   2> 290559 INFO  (SUITE-TestReqParamsAPI-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 290559 INFO  (Thread-305) [    ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 290560 INFO  (Thread-305) [    ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 290669 INFO  (SUITE-TestReqParamsAPI-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.c.ZkTestServer start zk server on port:54237
   [junit4]   2> 290691 INFO  (jetty-launcher-382-thread-1) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 290691 INFO  (jetty-launcher-382-thread-2) [    ] o.e.j.s.Server jetty-9.3.14.v20161028
   [junit4]   2> 290695 INFO  (jetty-launcher-382-thread-1) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3718130f{/solr,null,AVAILABLE}
   [junit4]   2> 290695 INFO  (jetty-launcher-382-thread-2) [    ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3684cc20{/solr,null,AVAILABLE}
   [junit4]   2> 290696 INFO  (jetty-launcher-382-thread-1) [    ] o.e.j.s.AbstractConnector Started ServerConnector@7666374b{HTTP/1.1,[http/1.1]}{127.0.0.1:54239}
   [junit4]   2> 290697 INFO  (jetty-launcher-382-thread-1) [    ] o.e.j.s.Server Started @295842ms
   [junit4]   2> 290697 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54239}
   [junit4]   2> 290697 ERROR (jetty-launcher-382-thread-1) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 290697 INFO  (jetty-launcher-382-thread-2) [    ] o.e.j.s.AbstractConnector Started ServerConnector@62849540{HTTP/1.1,[http/1.1]}{127.0.0.1:54240}
   [junit4]   2> 290697 INFO  (jetty-launcher-382-thread-2) [    ] o.e.j.s.Server Started @295843ms
   [junit4]   2> 290698 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54240}
   [junit4]   2> 290698 ERROR (jetty-launcher-382-thread-2) [    ] o.a.s.s.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 290698 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 290698 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 290698 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 290698 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-13T01:47:54.934Z
   [junit4]   2> 290699 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 6.5.0
   [junit4]   2> 290700 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 290700 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: null
   [junit4]   2> 290700 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2017-02-13T01:47:54.936Z
   [junit4]   2> 290707 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 290708 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 290720 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 290720 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.u.UpdateShardHandler Creating UpdateShardHandler HTTP client with params: socketTimeout=340000&connTimeout=45000&retry=true
   [junit4]   2> 290721 INFO  (jetty-launcher-382-thread-2) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54237/solr
   [junit4]   2> 290722 INFO  (jetty-launcher-382-thread-1) [    ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:54237/solr
   [junit4]   2> 290778 INFO  (jetty-launcher-382-thread-1) [n:127.0.0.1:54239_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 290778 INFO  (jetty-launcher-382-thread-2) [n:127.0.0.1:54240_solr    ] o.a.s.c.Overseer Overseer (id=null) closing
   [junit4]   2> 290779 INFO  (jetty-launcher-382-thread-1) [n:127.0.0.1:54239_solr    ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54239_solr
   [junit4]   2> 290781 INFO  (jetty-launcher-382-thread-2) [n:127.0.0.1:54240_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54240_solr
   [junit4]   2> 290782 INFO  (jetty-launcher-382-thread-1) [n:127.0.0.1:54239_solr    ] o.a.s.c.Overseer Overseer (id=97448786316361734-127.0.0.1:54239_solr-n_0000000000) starting
   [junit4]   2> 290792 INFO  (zkCallback-392-thread-1-processing-n:127.0.0.1:54240_solr) [n:127.0.0.1:54240_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 290792 INFO  (zkCallback-393-thread-1-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 290826 INFO  (jetty-launcher-382-thread-1) [n:127.0.0.1:54239_solr    ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54239_solr
   [junit4]   2> 290847 INFO  (zkCallback-393-thread-1-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 290848 INFO  (zkCallback-392-thread-1-processing-n:127.0.0.1:54240_solr) [n:127.0.0.1:54240_solr    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 290909 INFO  (jetty-launcher-382-thread-2) [n:127.0.0.1:54240_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/.
   [junit4]   2> 290931 INFO  (jetty-launcher-382-thread-1) [n:127.0.0.1:54239_solr    ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/.
   [junit4]   2> 290979 INFO  (SUITE-TestReqParamsAPI-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 290982 INFO  (SUITE-TestReqParamsAPI-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:54237/solr ready
   [junit4]   2> 291038 INFO  (qtp667814353-1718) [n:127.0.0.1:54240_solr    ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params replicationFactor=2&collection.configName=conf1&name=collection1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 291042 INFO  (OverseerThreadFactory-501-thread-1-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 291166 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica2&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2
   [junit4]   2> 291166 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr    ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica1&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2
   [junit4]   2> 291289 INFO  (zkCallback-393-thread-1-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 291290 INFO  (zkCallback-392-thread-1-processing-n:127.0.0.1:54240_solr) [n:127.0.0.1:54240_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 292190 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 292190 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 292253 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 292254 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 292258 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 292258 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 292260 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 292260 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 292261 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/./collection1_shard1_replica2/data/]
   [junit4]   2> 292261 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/./collection1_shard1_replica1/data/]
   [junit4]   2> 292339 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 292339 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 292339 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 292339 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 292341 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 292341 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 292341 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 292341 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 292342 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@8949e3f[collection1_shard1_replica1] main]
   [junit4]   2> 292342 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@3c98d8a8[collection1_shard1_replica2] main]
   [junit4]   2> 292344 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 292344 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 292345 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 292345 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 292364 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 292364 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 292374 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 292374 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 292374 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 292374 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 292374 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180582961807360
   [junit4]   2> 292374 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180582961807360
   [junit4]   2> 292376 INFO  (searcherExecutor-506-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@3c98d8a8[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 292376 INFO  (searcherExecutor-507-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@8949e3f[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 292384 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 292384 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 292384 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:54240/solr/collection1_shard1_replica2/
   [junit4]   2> 292385 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica2 url=http://127.0.0.1:54240/solr START replicas=[http://127.0.0.1:54239/solr/collection1_shard1_replica1/] nUpdates=100
   [junit4]   2> 292385 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica2 url=http://127.0.0.1:54240/solr DONE.  We have no versions.  sync failed.
   [junit4]   2> 292388 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 292389 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 292389 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 292395 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:54240/solr/collection1_shard1_replica2/ shard1
   [junit4]   2> 292509 INFO  (zkCallback-392-thread-1-processing-n:127.0.0.1:54240_solr) [n:127.0.0.1:54240_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 292509 INFO  (zkCallback-393-thread-2-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 292564 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 292567 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1  x:collection1_shard1_replica2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica2&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2} status=0 QTime=1400
   [junit4]   2> 292675 INFO  (zkCallback-393-thread-1-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 292675 INFO  (zkCallback-392-thread-1-processing-n:127.0.0.1:54240_solr) [n:127.0.0.1:54240_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 293390 INFO  (qtp1697556344-1723) [n:127.0.0.1:54239_solr c:collection1 s:shard1  x:collection1_shard1_replica1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=collection1_shard1_replica1&action=CREATE&numShards=1&collection=collection1&shard=shard1&wt=javabin&version=2} status=0 QTime=2223
   [junit4]   2> 293394 INFO  (qtp667814353-1718) [n:127.0.0.1:54240_solr    ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
   [junit4]   2> 293501 INFO  (zkCallback-393-thread-2-processing-n:127.0.0.1:54239_solr) [n:127.0.0.1:54239_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 293501 INFO  (zkCallback-392-thread-1-processing-n:127.0.0.1:54240_solr) [n:127.0.0.1:54240_solr    ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
   [junit4]   2> 294398 INFO  (qtp667814353-1718) [n:127.0.0.1:54240_solr    ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={replicationFactor=2&collection.configName=conf1&name=collection1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=3360
   [junit4]   2> 294417 INFO  (TEST-TestReqParamsAPI.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.SolrTestCaseJ4 ###Starting test
   [junit4]   2> 294476 INFO  (TEST-TestReqParamsAPI.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.c.TestSolrConfigHandler going to send config command. path /config?wt=json , payload: {
   [junit4]   2> 'create-requesthandler' : { 'name' : '/dump0', 'class': 'org.apache.solr.handler.DumpRequestHandler' }
   [junit4]   2> }
   [junit4]   2> 294519 INFO  (qtp1697556344-1727) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"create-requesthandler":{
   [junit4]   2>     "name":"/dump0",
   [junit4]   2>     "class":"org.apache.solr.handler.DumpRequestHandler"}}]
   [junit4]   2> 294521 INFO  (Thread-324) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 294522 INFO  (Thread-325) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 294523 INFO  (Thread-324) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 294523 INFO  (Thread-325) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 294526 INFO  (qtp1697556344-1727) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 2 replicas to set the property overlay to be of version 0 for collection collection1
   [junit4]   2> 294528 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294529 INFO  (solrHandlerExecutor-516-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294539 INFO  (Thread-324) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 294539 INFO  (Thread-325) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 294548 INFO  (Thread-324) [n:127.0.0.1:54240_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 294548 INFO  (Thread-325) [n:127.0.0.1:54239_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 294552 INFO  (Thread-324) [n:127.0.0.1:54240_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294554 INFO  (Thread-325) [n:127.0.0.1:54239_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294558 INFO  (Thread-324) [n:127.0.0.1:54240_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 294560 INFO  (Thread-325) [n:127.0.0.1:54239_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 294560 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/./collection1_shard1_replica2/data/]
   [junit4]   2> 294571 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/./collection1_shard1_replica1/data/]
   [junit4]   2> 294634 INFO  (qtp667814353-1721) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 294635 INFO  (qtp667814353-1721) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 294635 INFO  (qtp1697556344-1729) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 294636 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 294636 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 294636 INFO  (qtp1697556344-1729) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 294637 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 294637 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 294638 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:54239/solr/collection1_shard1_replica1/ for prop overlay   after 1 attempts
   [junit4]   2> 294638 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294639 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 294639 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 294640 INFO  (solrHandlerExecutor-516-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:54240/solr/collection1_shard1_replica2/ for prop overlay   after 1 attempts
   [junit4]   2> 294640 INFO  (solrHandlerExecutor-516-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294770 INFO  (qtp1697556344-1716) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 294770 INFO  (qtp1697556344-1716) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 294771 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 294771 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:54239/solr/collection1_shard1_replica1/ for prop overlay   after 2 attempts
   [junit4]   2> 294771 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294783 INFO  (qtp667814353-1728) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 294784 INFO  (qtp667814353-1728) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 294784 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 294784 INFO  (solrHandlerExecutor-516-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:54240/solr/collection1_shard1_replica2/ for prop overlay   after 2 attempts
   [junit4]   2> 294784 INFO  (solrHandlerExecutor-516-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294792 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 294792 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 294793 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@48d441fd[collection1_shard1_replica2] main]
   [junit4]   2> 294795 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 294796 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 294796 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 294797 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 294797 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 294798 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180585503555584
   [junit4]   2> 294798 INFO  (searcherExecutor-517-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@48d441fd[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 294821 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 294821 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 294849 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@637c4096[collection1_shard1_replica1] main]
   [junit4]   2> 294850 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 294851 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@1c10935b[collection1_shard1_replica2] main]
   [junit4]   2> 294851 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@2e1592e2
   [junit4]   2> 294851 INFO  (searcherExecutor-517-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@1c10935b[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 294852 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 294852 INFO  (Thread-324) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 294852 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 294852 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 294854 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 294855 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 294855 INFO  (searcherExecutor-518-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@637c4096[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 294855 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180585563324416
   [junit4]   2> 294864 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 294873 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 294878 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294880 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 294880 INFO  (qtp1697556344-1724) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 0 but my version is -1
   [junit4]   2> 294881 INFO  (qtp1697556344-1724) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 294881 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Another reload is in progress . Not doing anything
   [junit4]   2> 294881 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 0 from http://127.0.0.1:54239/solr/collection1_shard1_replica1/ for prop overlay   after 3 attempts
   [junit4]   2> 294881 INFO  (solrHandlerExecutor-516-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294890 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/./collection1_shard1_replica2/data/]
   [junit4]   2> 294903 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 294903 INFO  (Thread-325) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@33f1ce86[collection1_shard1_replica1] main]
   [junit4]   2> 294905 INFO  (searcherExecutor-518-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@33f1ce86[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 294909 INFO  (qtp667814353-1730) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
   [junit4]   2> 294909 INFO  (qtp667814353-1730) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 294909 INFO  (qtp667814353-1730) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 294925 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 294932 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 294933 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 294934 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 294935 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/./collection1_shard1_replica1/data/]
   [junit4]   2> 294986 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
   [junit4]   2> 294986 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 294986 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 294987 INFO  (qtp1697556344-1727) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Took 468.0ms to set the property overlay to be of version 0 for collection collection1
   [junit4]   2> 294988 INFO  (qtp1697556344-1727) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config params={wt=json} status=0 QTime=508
   [junit4]   2> 294988 INFO  (qtp1697556344-1727) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4df8a616
   [junit4]   2> 294988 INFO  (qtp1697556344-1727) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica1
   [junit4]   2> 294989 INFO  (TEST-TestReqParamsAPI.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.c.TestSolrConfigHandler going to send config command. path /config?wt=json , payload: {
   [junit4]   2> 'create-requesthandler' : { 'name' : '/dump1', 'class': 'org.apache.solr.handler.DumpRequestHandler', 'useParams':'x' }
   [junit4]   2> }
   [junit4]   2> 294993 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.ZkController Persisted config data to node /configs/conf1/configoverlay.json 
   [junit4]   2> 294993 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Executed config commands successfully and persisted to ZK [{"create-requesthandler":{
   [junit4]   2>     "name":"/dump1",
   [junit4]   2>     "class":"org.apache.solr.handler.DumpRequestHandler",
   [junit4]   2>     "useParams":"x"}}]
   [junit4]   2> 294994 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 2 replicas to set the property overlay to be of version 1 for collection collection1
   [junit4]   2> 294995 INFO  (Thread-327) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 294995 INFO  (Thread-326) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 294997 INFO  (Thread-327) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 294997 INFO  (Thread-326) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 294997 INFO  (solrHandlerExecutor-527-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 294997 INFO  (solrHandlerExecutor-527-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 295028 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 295028 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 295031 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 295031 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 295033 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@2e438a2f[collection1_shard1_replica1] main]
   [junit4]   2> 295034 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@186bee85[collection1_shard1_replica2] main]
   [junit4]   2> 295035 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 295036 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 295036 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 295036 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 295036 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 295036 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 295037 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 295038 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 295038 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 295038 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 295038 INFO  (searcherExecutor-525-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@186bee85[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295038 INFO  (searcherExecutor-526-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@2e438a2f[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295038 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180585755213824
   [junit4]   2> 295038 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180585755213824
   [junit4]   2> 295041 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 295042 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 295042 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6013cd09[collection1_shard1_replica1] main]
   [junit4]   2> 295042 INFO  (searcherExecutor-526-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@6013cd09[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295052 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@bd64af7[collection1_shard1_replica2] main]
   [junit4]   2> 295053 INFO  (OverseerCollectionConfigSetProcessor-97448786316361734-127.0.0.1:54239_solr-n_0000000000) [n:127.0.0.1:54239_solr    ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 295057 INFO  (Thread-326) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 295064 INFO  (Thread-326) [n:127.0.0.1:54239_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 295064 INFO  (searcherExecutor-525-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@bd64af7[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295065 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@5fcf2d7b
   [junit4]   2> 295067 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 295067 INFO  (Thread-326) [n:127.0.0.1:54239_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 295070 INFO  (Thread-326) [n:127.0.0.1:54239_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 295075 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [[collection1_shard1_replica1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/collection1_shard1_replica1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node1/./collection1_shard1_replica1/data/]
   [junit4]   2> 295076 INFO  (Thread-327) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 295085 INFO  (Thread-327) [n:127.0.0.1:54240_solr    ] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 295088 INFO  (Thread-327) [n:127.0.0.1:54240_solr    ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 295089 INFO  (Thread-327) [n:127.0.0.1:54240_solr    ] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 295100 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/./collection1_shard1_replica2/data/]
   [junit4]   2> 295101 INFO  (qtp1697556344-1724) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 295103 INFO  (qtp667814353-1715) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler expecting overlay version 1 but my version is 0
   [junit4]   2> 295103 INFO  (qtp1697556344-1724) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=2
   [junit4]   2> 295103 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 295103 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 295104 INFO  (qtp667814353-1715) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 295105 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler Trying to update my configs
   [junit4]   2> 295105 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 295105 INFO  (solrHandlerExecutor-527-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:54239/solr/collection1_shard1_replica1/ for prop overlay   after 1 attempts
   [junit4]   2> 295105 INFO  (solrHandlerExecutor-527-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 295106 INFO  (solrHandlerExecutor-527-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Could not get expectedVersion 1 from http://127.0.0.1:54240/solr/collection1_shard1_replica2/ for prop overlay   after 1 attempts
   [junit4]   2> 295106 INFO  (solrHandlerExecutor-527-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 295107 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore core reload collection1_shard1_replica1
   [junit4]   2> 295108 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore core reload collection1_shard1_replica2
   [junit4]   2> 295163 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 295163 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 295165 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@6eb3e086[collection1_shard1_replica2] main]
   [junit4]   2> 295167 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 295167 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 295168 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 295168 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 295168 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 295169 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180585892577280
   [junit4]   2> 295169 INFO  (searcherExecutor-535-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@6eb3e086[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295179 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
   [junit4]   2> 295179 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 295180 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 295181 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.SolrIndexSearcher Opening [Searcher@2d83e285[collection1_shard1_replica2] main]
   [junit4]   2> 295181 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2]  CLOSING SolrCore org.apache.solr.core.SolrCore@412685b
   [junit4]   2> 295181 INFO  (searcherExecutor-535-thread-1-processing-n:127.0.0.1:54240_solr x:collection1_shard1_replica2 s:shard1 c:collection1 r:core_node2) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [collection1_shard1_replica2] Registered new searcher Searcher@2d83e285[collection1_shard1_replica2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295182 INFO  (Thread-327) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica2
   [junit4]   2> 295182 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@66775be9[collection1_shard1_replica1] main]
   [junit4]   2> 295184 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 295185 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 295185 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/conf1/managed-schema
   [junit4]   2> 295186 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 295186 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.ReplicationHandler Commits will be reserved for  10000
   [junit4]   2> 295186 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1559180585910403072
   [junit4]   2> 295186 INFO  (searcherExecutor-534-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@66775be9[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295195 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 295206 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
   [junit4]   2> 295206 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.SolrIndexSearcher Opening [Searcher@32e0e5cd[collection1_shard1_replica1] main]
   [junit4]   2> 295207 INFO  (searcherExecutor-534-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1] Registered new searcher Searcher@32e0e5cd[collection1_shard1_replica1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 295207 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@481b85c9
   [junit4]   2> 295209 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler I already have the expected version 1 of config
   [junit4]   2> 295209 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 295209 INFO  (qtp1697556344-1725) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 295210 INFO  (Thread-326) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica1
   [junit4]   2> 295210 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema [collection1_shard1_replica2] Schema name=minimal
   [junit4]   2> 295212 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler I already have the expected version 1 of config
   [junit4]   2> 295212 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
   [junit4]   2> 295212 INFO  (qtp667814353-1720) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.S.Request [collection1_shard1_replica2]  webapp=/solr path=/config/znodeVersion params={overlay=1&wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 295214 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Took 220.0ms to set the property overlay to be of version 1 for collection collection1
   [junit4]   2> 295214 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.S.Request [collection1_shard1_replica1]  webapp=/solr path=/config params={wt=json} status=0 QTime=222
   [junit4]   2> 295214 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrCore [collection1_shard1_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@36a69675
   [junit4]   2> 295214 INFO  (qtp1697556344-1726) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1.shard1.replica1
   [junit4]   2> 295215 INFO  (TEST-TestReqParamsAPI.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):90
   [junit4]   2> 295216 INFO  (TEST-TestReqParamsAPI.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
   [junit4]   2> 295216 INFO  (TEST-TestReqParamsAPI.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.c.TestSolrConfigHandler going to send config command. path /config/params?wt=json , payload:  {
   [junit4]   2>   'set' : {'x': {                    'a':'A val',
   [junit4]   2>                     'b': 'B val'}
   [junit4]   2>              }
   [junit4]   2>   }
   [junit4]   2> 295216 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 295219 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica2' using configuration from collection collection1
   [junit4]   2> 295221 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 6.5.0
   [junit4]   2> 295227 INFO  (qtp1697556344-1729) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Waiting up to 30 secs for 2 replicas to set the property params to be of version 0 for collection collection1
   [junit4]   2> 295228 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema [collection1_shard1_replica1] Schema name=minimal
   [junit4]   2> 295229 INFO  (Thread-329) [n:127.0.0.1:54240_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica2
   [junit4]   2> 295230 INFO  (Thread-328) [n:127.0.0.1:54239_solr    ] o.a.s.c.SolrCore config update listener called for core collection1_shard1_replica1
   [junit4]   2> 295231 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 295232 INFO  (solrHandlerExecutor-542-thread-1-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 295232 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.c.CoreContainer Reloading SolrCore 'collection1_shard1_replica1' using configuration from collection collection1
   [junit4]   2> 295232 INFO  (Thread-329) [n:127.0.0.1:54240_solr    ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0 
   [junit4]   2> 295232 INFO  (solrHandlerExecutor-542-thread-2-processing-n:127.0.0.1:54239_solr x:collection1_shard1_replica1 s:shard1 c:collection1 r:core_node1) [n:127.0.0.1:54239_solr c:collection1 s:shard1 r:core_node1 x:collection1_shard1_replica1] o.a.s.h.SolrConfigHandler Time elapsed : 0 secs, maxWait 30
   [junit4]   2> 295233 INFO  (Thread-329) [n:127.0.0.1:54240_solr    ] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 295233 INFO  (Thread-328) [n:127.0.0.1:54239_solr    ] o.a.s.c.RequestParams conf resource params.json loaded . version : 0 
   [junit4]   2> 295233 INFO  (Thread-328) [n:127.0.0.1:54239_solr    ] o.a.s.c.RequestParams request params refreshed to version 0
   [junit4]   2> 295240 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54240_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica2] o.a.s.c.SolrCore [[collection1_shard1_replica2] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.handler.TestReqParamsAPI_3CF871AE4C014DE1-001/tempDir-001/node2/collection1_shard1_replica2], dataDir=[/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J0/temp/solr.ha

[...truncated too long message...]

.0.0.1:60718_m_c%2Fbi) [n:127.0.0.1:60718_m_c%2Fbi c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.c.SolrCore [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@24830bc8
   [junit4]   2> 2595571 INFO  (zkCallback-3506-thread-1-processing-n:127.0.0.1:60718_m_c%2Fbi) [n:127.0.0.1:60718_m_c%2Fbi c:collection1 s:shard1 r:core_node4 x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.core.collection1
   [junit4]   2> 2595572 INFO  (TEST-MultiThreadedOCPTest.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.m.SolrMetricManager Closing metric reporters for: solr.node
   [junit4]   2> 2595572 INFO  (TEST-MultiThreadedOCPTest.test-seed#[3CF871AE4C014DE1]) [    ] o.e.j.s.AbstractConnector Stopped ServerConnector@7527118f{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
   [junit4]   2> 2595573 INFO  (TEST-MultiThreadedOCPTest.test-seed#[3CF871AE4C014DE1]) [    ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5662564e{/m_c/bi,null,UNAVAILABLE}
   [junit4]   2> 2595574 INFO  (TEST-MultiThreadedOCPTest.test-seed#[3CF871AE4C014DE1]) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60692 60692
   [junit4]   2> 2595598 INFO  (Thread-3906) [    ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:60692 60692
   [junit4]   2> 2602003 WARN  (Thread-3906) [    ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/aliases.json
   [junit4]   2> 	6	/solr/clusterprops.json
   [junit4]   2> 	5	/solr/security.json
   [junit4]   2> 	5	/solr/configs/conf1
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	6	/solr/clusterstate.json
   [junit4]   2> 	4	/solr/collections/ocptest_shardsplit2/state.json
   [junit4]   2> 	4	/solr/collections/ocptest1/state.json
   [junit4]   2> 	4	/solr/collections/collection1/state.json
   [junit4]   2> 	4	/solr/collections/ocptest3/state.json
   [junit4]   2> 	4	/solr/collections/ocptest2/state.json
   [junit4]   2> 	4	/solr/collections/ocptest4/state.json
   [junit4]   2> 	4	/solr/collections/ocptest_shardsplit/state.json
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	77	/solr/overseer/queue
   [junit4]   2> 	23	/solr/overseer/queue-work
   [junit4]   2> 	19	/solr/overseer/collection-queue-work
   [junit4]   2> 	6	/solr/live_nodes
   [junit4]   2> 	6	/solr/collections
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=MultiThreadedOCPTest -Dtests.method=test -Dtests.seed=3CF871AE4C014DE1 -Dtests.slow=true -Dtests.locale=en-PH -Dtests.timezone=Antarctica/DumontDUrville -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] FAILURE 46.5s J1 | MultiThreadedOCPTest.test <<<
   [junit4]    > Throwable #1: java.lang.AssertionError: Task 3001 did not complete, final state: FAILED expected same:<COMPLETED> was not:<FAILED>
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([3CF871AE4C014DE1:B4AC4E74E2FD2019]:0)
   [junit4]    > 	at org.apache.solr.cloud.MultiThreadedOCPTest.testDeduplicationOfSubmittedTasks(MultiThreadedOCPTest.java:250)
   [junit4]    > 	at org.apache.solr.cloud.MultiThreadedOCPTest.test(MultiThreadedOCPTest.java:64)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
   [junit4]    > 	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
   [junit4]    > 	at java.lang.Thread.run(Thread.java:745)
   [junit4]   2> 2602137 INFO  (SUITE-MultiThreadedOCPTest-seed#[3CF871AE4C014DE1]-worker) [    ] o.a.s.SolrTestCaseJ4 ###deleteCore
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.MultiThreadedOCPTest_3CF871AE4C014DE1-001
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): {}, docValues:{}, maxPointsInLeafNode=1024, maxMBSortInHeap=5.6140785014371115, sim=RandomSimilarity(queryNorm=false,coord=crazy): {}, locale=en-PH, timezone=Antarctica/DumontDUrville
   [junit4]   2> NOTE: Mac OS X 10.11.6 x86_64/Oracle Corporation 1.8.0_121 (64-bit)/cpus=3,threads=1,free=160295360,total=512229376
   [junit4]   2> NOTE: All tests run in this JVM: [TestCloudPseudoReturnFields, RulesTest, SolrInfoMBeanTest, TestStandardQParsers, TestJoin, TestQuerySenderListener, BadIndexSchemaTest, SecurityConfHandlerTest, TestTolerantUpdateProcessorRandomCloud, TolerantUpdateProcessorTest, TestNRTOpen, TestConfigOverlay, TestValueSourceCache, SpatialRPTFieldTypeTest, TestDocSet, TestReversedWildcardFilterFactory, TestDynamicLoading, OverseerStatusTest, CollectionStateFormat2Test, SortSpecParsingTest, TestClusterProperties, ScriptEngineTest, TestFuzzyAnalyzedSuggestions, TestCoreAdminApis, PKIAuthenticationIntegrationTest, JsonLoaderTest, TermVectorComponentTest, QueryResultKeyTest, TestQueryWrapperFilter, BJQParserTest, TestSolrCloudWithDelegationTokens, CoreSorterTest, RegexBoostProcessorTest, TestLMDirichletSimilarityFactory, UpdateRequestProcessorFactoryTest, RemoteQueryErrorTest, DeleteReplicaTest, TestFilteredDocIdSet, TestSolrDeletionPolicy2, ChangedSchemaMergeTest, LoggingHandlerTest, TestMiniSolrCloudCluster, TestQueryTypes, TestStressLiveNodes, AssignTest, TestQueryUtils, BinaryUpdateRequestHandlerTest, ExternalFileFieldSortTest, TestObjectReleaseTracker, AnalyticsQueryTest, HdfsBasicDistributedZk2Test, IndexSchemaTest, DebugComponentTest, FullSolrCloudDistribCmdsTest, BaseCdcrDistributedZkTest, BasicZkTest, SignatureUpdateProcessorFactoryTest, SuggestComponentTest, TestBlendedInfixSuggestions, SpellPossibilityIteratorTest, HdfsRestartWhileUpdatingTest, TestSimpleTrackingShardHandler, TestReload, VersionInfoTest, TestNoOpRegenerator, LeaderElectionIntegrationTest, TestFieldCache, RequestLoggingTest, TestDeleteCollectionOnDownNodes, SmileWriterTest, TestOmitPositions, SuggesterWFSTTest, DistributedFacetPivotLargeTest, TestPathTrie, MigrateRouteKeyTest, SolrGraphiteReporterTest, TestScoreJoinQPScore, TestRebalanceLeaders, TestSlowCompositeReaderWrapper, TestMacros, TemplateUpdateProcessorTest, SolrIndexConfigTest, DocValuesNotIndexedTest, ZkSolrClientTest, TestJsonFacets, RequestHandlersTest, TestDistributedMissingSort, CurrencyFieldOpenExchangeTest, TestZkChroot, CopyFieldTest, RAMDirectoryFactoryTest, CloneFieldUpdateProcessorFactoryTest, TestRealTimeGet, TestImpersonationWithHadoopAuth, TestHashQParserPlugin, ActionThrottleTest, OverriddenZkACLAndCredentialsProvidersTest, PreAnalyzedUpdateProcessorTest, TestJavabinTupleStreamParser, CloudExitableDirectoryReaderTest, HLLSerializationTest, BigEndianAscendingWordSerializerTest, TriLevelCompositeIdRoutingTest, TestFiltering, BasicDistributedZk2Test, TestCloudSchemaless, SolrSlf4jReporterTest, SearchHandlerTest, SortByFunctionTest, ConcurrentDeleteAndCreateCollectionTest, BadCopyFieldTest, TestNonDefinedSimilarityFactory, HdfsRecoverLeaseTest, DistributedExpandComponentTest, TestShardHandlerFactory, EchoParamsTest, AtomicUpdatesTest, BlockCacheTest, TestRequestForwarding, TestXmlQParserPlugin, TestRestoreCore, DefaultValueUpdateProcessorTest, SpellCheckCollatorTest, TestBulkSchemaAPI, TestFieldTypeCollectionResource, TestConfigReload, DataDrivenBlockJoinTest, TestDownShardTolerantSearch, TestFoldingMultitermQuery, SolrMetricManagerTest, FieldMutatingUpdateProcessorTest, DateMathParserTest, TestSchemalessBufferedUpdates, TestUseDocValuesAsStored2, TestMiniSolrCloudClusterSSL, TestDistributedGrouping, PreAnalyzedFieldTest, TestCollectionAPIs, TestCopyFieldCollectionResource, TestSearcherReuse, TestLuceneMatchVersion, TestSubQueryTransformerDistrib, SolrIndexMetricsTest, TestManagedSchema, ShowFileRequestHandlerTest, TestLMJelinekMercerSimilarityFactory, DistribDocExpirationUpdateProcessorTest, ExplicitHLLTest, TestSolrCoreParser, TestBadConfig, TestUninvertingReader, TestDefaultStatsCache, HdfsUnloadDistributedZkTest, PeerSyncReplicationTest, OpenCloseCoreStressTest, TestSQLHandlerNonCloud, HighlighterConfigTest, ClusterStateTest, DistanceUnitsTest, ReturnFieldsTest, BlockJoinFacetRandomTest, TestConfigSetsAPIZkFailure, TestMergePolicyConfig, GraphQueryTest, FieldAnalysisRequestHandlerTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, CollectionsAPIDistributedZkTest, RecoveryZkTest, TestReplicationHandler, TestRandomFaceting, TestRandomDVFaceting, ZkCLITest, ShardRoutingCustomTest, AutoCommitTest, TestFaceting, TestRecovery, TermVectorComponentDistributedTest, ZkControllerTest, TestStressReorder, TestStressVersions, HardAutoCommitTest, TestRangeQuery, StatsComponentTest, SolrCmdDistributorTest, DirectUpdateHandlerTest, HighlighterTest, CurrencyFieldXmlFileTest, SolrIndexSplitterTest, SimplePostToolTest, TestCoreDiscovery, CoreAdminHandlerTest, SuggesterTSTTest, SuggesterTest, TestStressLucene, TestTrie, SpatialFilterTest, TestCSVLoader, PolyFieldTest, NoCacheHeaderTest, WordBreakSolrSpellCheckerTest, TestAtomicUpdateErrorCases, TestWordDelimiterFilterFactory, DistanceFunctionTest, TestSolrDeletionPolicy1, IndexBasedSpellCheckerTest, LukeRequestHandlerTest, DirectSolrSpellCheckerTest, TermsComponentTest, TestIndexingPerformance, RequiredFieldsTest, FastVectorHighlighterTest, TestJmxIntegration, QueryParsingTest, TestPHPSerializedResponseWriter, TestLFUCache, TestPerFieldSimilarity, TestFastWriter, ResourceLoaderTest, URLClassifyProcessorTest, ChaosMonkeyNothingIsSafeTest, CdcrReplicationDistributedZkTest, CdcrRequestHandlerTest, CdcrVersionReplicationTest, CollectionTooManyReplicasTest, CollectionsAPISolrJTest, CustomCollectionTest, DeleteShardTest, DistribCursorPagingTest, HttpPartitionTest, LeaderFailureAfterFreshStartTest, MultiThreadedOCPTest]
   [junit4] Completed [499/691 (2!)] on J1 in 46.53s, 1 test, 1 failure <<< FAILURES!

[...truncated 63278 lines...]


[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 692 - Still Failing!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/692/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 75397 lines...]
-documentation-lint:
    [jtidy] Checking for broken html (such as invalid tags)...
   [delete] Deleting directory /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/build/jtidy_tmp
     [echo] Checking for broken links...
     [exec] 
     [exec] Crawl/parse...
     [exec] 
     [exec] Verify...
     [echo] Checking for malformed docs...
     [exec] 
     [exec] /Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/docs/solr-solrj/overview-summary.html
     [exec]   missing: org.apache.solr.client.solrj.io.eval
     [exec] 
     [exec] Missing javadocs were found!

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:775: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:101: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:654: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:670: The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:2490: exec returned: 1

Total time: 119 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any