You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2020/05/23 05:42:24 UTC

[JENKINS] Lucene-Solr-8.x-MacOSX (64bit/jdk-11.0.6) - Build # 1033 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/1033/
Java: 64bit/jdk-11.0.6 -XX:-UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.MultiSolrCloudTestCaseTest

Error Message:
2 threads leaked from SUITE scope at org.apache.solr.cloud.MultiSolrCloudTestCaseTest:     1) Thread[id=888, name=SessionTracker, state=TIMED_WAITING, group=TGRP-MultiSolrCloudTestCaseTest]         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)    2) Thread[id=821, name=SessionTracker, state=TIMED_WAITING, group=TGRP-MultiSolrCloudTestCaseTest]         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.MultiSolrCloudTestCaseTest: 
   1) Thread[id=888, name=SessionTracker, state=TIMED_WAITING, group=TGRP-MultiSolrCloudTestCaseTest]
        at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
   2) Thread[id=821, name=SessionTracker, state=TIMED_WAITING, group=TGRP-MultiSolrCloudTestCaseTest]
        at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
	at __randomizedtesting.SeedInfo.seed([36D25A3BB9B293B3]:0)


FAILED:  org.apache.solr.cloud.ReplaceNodeTest.test

Error Message:
Expecting no cores but found some: {replacenodetest_coll_shard1_replica_n21={name=replacenodetest_coll_shard1_replica_n21,instanceDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard1_replica_n21,dataDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard1_replica_n21/data/,config=solrconfig.xml,schema=schema.xml,startTime=Sat May 23 08:54:51 RET 2020,uptime=20892,lastPublished=active,configVersion=0,cloud={collection=replacenodetest_coll,shard=shard1,replica=core_node22,replicaType=NRT},index={numDocs=0,maxDoc=0,deletedDocs=0,indexHeapUsageBytes=0,version=3,segmentCount=0,current=false,hasDeletions=false,directory=org.apache.lucene.store.MockDirectoryWrapper:MockDirectoryWrapper(RAMDirectory@755c80e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11a59a8e),segmentsFile=segments_1,segmentsFileSizeInBytes=-1,userData={},sizeInBytes=135,size=135 bytes}},replacenodetest_coll_shard5_replica_n23={name=replacenodetest_coll_shard5_replica_n23,instanceDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard5_replica_n23,dataDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard5_replica_n23/data/,config=solrconfig.xml,schema=schema.xml,startTime=Sat May 23 08:54:52 RET 2020,uptime=19374,lastPublished=active,configVersion=0,cloud={collection=replacenodetest_coll,shard=shard5,replica=core_node24,replicaType=NRT},index={numDocs=0,maxDoc=0,deletedDocs=0,indexHeapUsageBytes=0,version=3,segmentCount=0,current=false,hasDeletions=false,directory=org.apache.lucene.store.MockDirectoryWrapper:MockDirectoryWrapper(RAMDirectory@6c46d452 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6526595a),segmentsFile=segments_1,segmentsFileSizeInBytes=-1,userData={},sizeInBytes=135,size=135 bytes}}} expected:<0> but was:<2>

Stack Trace:
java.lang.AssertionError: Expecting no cores but found some: {replacenodetest_coll_shard1_replica_n21={name=replacenodetest_coll_shard1_replica_n21,instanceDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard1_replica_n21,dataDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard1_replica_n21/data/,config=solrconfig.xml,schema=schema.xml,startTime=Sat May 23 08:54:51 RET 2020,uptime=20892,lastPublished=active,configVersion=0,cloud={collection=replacenodetest_coll,shard=shard1,replica=core_node22,replicaType=NRT},index={numDocs=0,maxDoc=0,deletedDocs=0,indexHeapUsageBytes=0,version=3,segmentCount=0,current=false,hasDeletions=false,directory=org.apache.lucene.store.MockDirectoryWrapper:MockDirectoryWrapper(RAMDirectory@755c80e8 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@11a59a8e),segmentsFile=segments_1,segmentsFileSizeInBytes=-1,userData={},sizeInBytes=135,size=135 bytes}},replacenodetest_coll_shard5_replica_n23={name=replacenodetest_coll_shard5_replica_n23,instanceDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard5_replica_n23,dataDir=/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001/tempDir-001/node1/./replacenodetest_coll_shard5_replica_n23/data/,config=solrconfig.xml,schema=schema.xml,startTime=Sat May 23 08:54:52 RET 2020,uptime=19374,lastPublished=active,configVersion=0,cloud={collection=replacenodetest_coll,shard=shard5,replica=core_node24,replicaType=NRT},index={numDocs=0,maxDoc=0,deletedDocs=0,indexHeapUsageBytes=0,version=3,segmentCount=0,current=false,hasDeletions=false,directory=org.apache.lucene.store.MockDirectoryWrapper:MockDirectoryWrapper(RAMDirectory@6c46d452 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@6526595a),segmentsFile=segments_1,segmentsFileSizeInBytes=-1,userData={},sizeInBytes=135,size=135 bytes}}} expected:<0> but was:<2>
	at __randomizedtesting.SeedInfo.seed([36D25A3BB9B293B3:BE8665E1174EFE4B]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.apache.solr.cloud.ReplaceNodeTest.test(ReplaceNodeTest.java:142)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.handler.admin.DaemonStreamApiTest.testAPIs

Error Message:
State for daemon 'daemon0' did not become TERMINATED in 10 seconds

Stack Trace:
java.lang.AssertionError: State for daemon 'daemon0' did not become TERMINATED in 10 seconds
	at __randomizedtesting.SeedInfo.seed([36D25A3BB9B293B3:34FF827074E8EA09]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.solr.handler.admin.DaemonStreamApiTest.checkStopped(DaemonStreamApiTest.java:226)
	at org.apache.solr.handler.admin.DaemonStreamApiTest.testAPIs(DaemonStreamApiTest.java:138)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.admin.DaemonStreamApiTest

Error Message:
4 threads leaked from SUITE scope at org.apache.solr.handler.admin.DaemonStreamApiTest:     1) Thread[id=1094, name=DaemonStream-daemon2-654-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2-EventThread, state=WAITING, group=TGRP-DaemonStreamApiTest]         at java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)         at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)         at java.base@11.0.6/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)    2) Thread[id=963, name=DaemonStream-daemon1-652-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2, state=BLOCKED, group=TGRP-DaemonStreamApiTest]         at app//org.apache.solr.client.solrj.io.SolrClientCache.getCloudSolrClient(SolrClientCache.java:57)         at app//org.apache.solr.client.solrj.io.stream.UpdateStream.setCloudSolrClient(UpdateStream.java:290)         at app//org.apache.solr.client.solrj.io.stream.UpdateStream.open(UpdateStream.java:115)         at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.stream(DaemonStream.java:335)         at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.run(DaemonStream.java:319)         at java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)         at java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$209/0x000000014cb9b1b8.run(Unknown Source)         at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)         at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)    3) Thread[id=965, name=DaemonStream-daemon2-654-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2, state=TIMED_WAITING, group=TGRP-DaemonStreamApiTest]         at java.base@11.0.6/java.lang.Object.wait(Native Method)         at app//org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:243)         at app//org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:194)         at app//org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:127)         at app//org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)         at app//org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:345)         at app//org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:175)         at app//org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160)         at app//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.connect(BaseCloudSolrClient.java:335)         at app//org.apache.solr.client.solrj.io.SolrClientCache.getCloudSolrClient(SolrClientCache.java:67)         at app//org.apache.solr.client.solrj.io.stream.UpdateStream.setCloudSolrClient(UpdateStream.java:290)         at app//org.apache.solr.client.solrj.io.stream.UpdateStream.open(UpdateStream.java:115)         at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.stream(DaemonStream.java:335)         at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.run(DaemonStream.java:319)         at java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)         at java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$209/0x000000014cb9b1b8.run(Unknown Source)         at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)         at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)         at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)    4) Thread[id=1093, name=DaemonStream-daemon2-654-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2-SendThread(127.0.0.1:53734), state=TIMED_WAITING, group=TGRP-DaemonStreamApiTest]         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:369)         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 4 threads leaked from SUITE scope at org.apache.solr.handler.admin.DaemonStreamApiTest: 
   1) Thread[id=1094, name=DaemonStream-daemon2-654-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2-EventThread, state=WAITING, group=TGRP-DaemonStreamApiTest]
        at java.base@11.0.6/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.6/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.6/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.6/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
   2) Thread[id=963, name=DaemonStream-daemon1-652-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2, state=BLOCKED, group=TGRP-DaemonStreamApiTest]
        at app//org.apache.solr.client.solrj.io.SolrClientCache.getCloudSolrClient(SolrClientCache.java:57)
        at app//org.apache.solr.client.solrj.io.stream.UpdateStream.setCloudSolrClient(UpdateStream.java:290)
        at app//org.apache.solr.client.solrj.io.stream.UpdateStream.open(UpdateStream.java:115)
        at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.stream(DaemonStream.java:335)
        at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.run(DaemonStream.java:319)
        at java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$209/0x000000014cb9b1b8.run(Unknown Source)
        at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
   3) Thread[id=965, name=DaemonStream-daemon2-654-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2, state=TIMED_WAITING, group=TGRP-DaemonStreamApiTest]
        at java.base@11.0.6/java.lang.Object.wait(Native Method)
        at app//org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:243)
        at app//org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:194)
        at app//org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:127)
        at app//org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:117)
        at app//org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:345)
        at app//org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:175)
        at app//org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160)
        at app//org.apache.solr.client.solrj.impl.BaseCloudSolrClient.connect(BaseCloudSolrClient.java:335)
        at app//org.apache.solr.client.solrj.io.SolrClientCache.getCloudSolrClient(SolrClientCache.java:67)
        at app//org.apache.solr.client.solrj.io.stream.UpdateStream.setCloudSolrClient(UpdateStream.java:290)
        at app//org.apache.solr.client.solrj.io.stream.UpdateStream.open(UpdateStream.java:115)
        at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.stream(DaemonStream.java:335)
        at app//org.apache.solr.client.solrj.io.stream.DaemonStream$StreamRunner.run(DaemonStream.java:319)
        at java.base@11.0.6/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base@11.0.6/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$209/0x000000014cb9b1b8.run(Unknown Source)
        at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.6/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.6/java.lang.Thread.run(Thread.java:834)
   4) Thread[id=1093, name=DaemonStream-daemon2-654-thread-1-processing-n:127.0.0.1:53739_solr x:checkpointColl_shard1_replica_n1 c:checkpointColl s:shard1 r:core_node2-SendThread(127.0.0.1:53734), state=TIMED_WAITING, group=TGRP-DaemonStreamApiTest]
        at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:369)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1137)
	at __randomizedtesting.SeedInfo.seed([36D25A3BB9B293B3]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.handler.admin.HealthCheckHandlerTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.handler.admin.HealthCheckHandlerTest:     1) Thread[id=106, name=SessionTracker, state=TIMED_WAITING, group=TGRP-HealthCheckHandlerTest]         at java.base@11.0.6/java.lang.Thread.sleep(Native Method)         at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.handler.admin.HealthCheckHandlerTest: 
   1) Thread[id=106, name=SessionTracker, state=TIMED_WAITING, group=TGRP-HealthCheckHandlerTest]
        at java.base@11.0.6/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
	at __randomizedtesting.SeedInfo.seed([36D25A3BB9B293B3]:0)


FAILED:  org.apache.solr.update.processor.DistributedUpdateProcessorTest.testVersionAdd

Error Message:
 Expected: is <1>      but: was <2>

Stack Trace:
java.lang.AssertionError: 
Expected: is <1>
     but: was <2>
	at __randomizedtesting.SeedInfo.seed([36D25A3BB9B293B3:9E38A9C64EA27376]:0)
	at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
	at org.junit.Assert.assertThat(Assert.java:956)
	at org.junit.Assert.assertThat(Assert.java:923)
	at org.apache.solr.update.processor.DistributedUpdateProcessorTest.testVersionAdd(DistributedUpdateProcessorTest.java:107)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
	at org.mockito.internal.junit.JUnitRule$1.evaluateSafely(JUnitRule.java:52)
	at org.mockito.internal.junit.JUnitRule$1.evaluate(JUnitRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
	at java.base/java.lang.Thread.run(Thread.java:834)




Build Log:
[...truncated 13481 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.HealthCheckHandlerTest
   [junit4]   2> 61396 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/server/solr/configsets/_default/conf'
   [junit4]   2> 61396 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 61397 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.SolrTestCaseJ4 Created dataDir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J4/temp/solr.handler.admin.HealthCheckHandlerTest_36D25A3BB9B293B3-001/data-dir-6-001
   [junit4]   2> 61397 WARN  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
   [junit4]   2> 61397 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 61399 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0) w/ MAC_OS_X supressed clientAuth
   [junit4]   2> 61423 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 1 servers in /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J4/temp/solr.handler.admin.HealthCheckHandlerTest_36D25A3BB9B293B3-001/tempDir-001
   [junit4]   2> 61434 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 61437 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer client port: 0.0.0.0/0.0.0.0:0
   [junit4]   2> 61438 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 61540 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer start zk server on port: 53639
   [junit4]   2> 61541 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:53639
   [junit4]   2> 61542 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:53639
   [junit4]   2> 61542 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 53639
   [junit4]   2> 61640 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61678 INFO  (zkConnectionManagerCallback-115-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61678 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 61707 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61715 INFO  (zkConnectionManagerCallback-117-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61716 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 61718 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 61731 INFO  (zkConnectionManagerCallback-119-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 61732 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62175 WARN  (jetty-launcher-120-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 62176 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 62176 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
   [junit4]   2> 62189 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.Server jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 11.0.6+10
   [junit4]   2> 62209 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 62209 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 62213 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 62241 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@4dd49f44{/solr,null,AVAILABLE}
   [junit4]   2> 62248 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@27ecdff9{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:53644}
   [junit4]   2> 62248 INFO  (jetty-launcher-120-thread-1) [     ] o.e.j.s.Server Started @62326ms
   [junit4]   2> 62249 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53644}
   [junit4]   2> 62303 ERROR (jetty-launcher-120-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 62303 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 62304 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 62304 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 62304 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 62304 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-05-23T04:49:26.126941Z
   [junit4]   2> 62307 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62314 INFO  (zkConnectionManagerCallback-122-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62314 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62318 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 62585 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 62588 WARN  (jetty-launcher-120-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@318c055a[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 62588 WARN  (jetty-launcher-120-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@318c055a[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 62595 WARN  (jetty-launcher-120-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@7042f003[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 62595 WARN  (jetty-launcher-120-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@7042f003[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 62596 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53639/solr
   [junit4]   2> 62610 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62626 INFO  (zkConnectionManagerCallback-133-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62626 INFO  (jetty-launcher-120-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 62743 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 62748 INFO  (zkConnectionManagerCallback-135-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 62748 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 63133 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:53644_solr
   [junit4]   2> 63136 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.Overseer Overseer (id=72089327819948037-127.0.0.1:53644_solr-n_0000000000) starting
   [junit4]   2> 63249 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53644_solr
   [junit4]   2> 63250 INFO  (OverseerStateUpdate-72089327819948037-127.0.0.1:53644_solr-n_0000000000) [n:127.0.0.1:53644_solr     ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:53644_solr
   [junit4]   2> 63268 INFO  (zkCallback-134-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 63314 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 63325 WARN  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 63374 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 63429 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53644.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 63429 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53644.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 63430 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53644.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 63433 INFO  (jetty-launcher-120-thread-1) [n:127.0.0.1:53644_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J4/temp/solr.handler.admin.HealthCheckHandlerTest_36D25A3BB9B293B3-001/tempDir-001/node1/.
   [junit4]   2> 63490 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=1
   [junit4]   2> 63493 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 63499 INFO  (zkConnectionManagerCallback-152-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 63499 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 63503 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 63508 INFO  (SUITE-HealthCheckHandlerTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:53639/solr ready
   [junit4]   2> 63538 INFO  (TEST-HealthCheckHandlerTest.testFindUnhealthyCores-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testFindUnhealthyCores
   [junit4]   2> {
   [junit4]   2>   "collection1":{
   [junit4]   2>     "replicationFactor":"1",
   [junit4]   2>     "maxShardsPerNode":"1",
   [junit4]   2>     "shards":{"slice1":{
   [junit4]   2>         "state":"active",
   [junit4]   2>         "replicas":{
   [junit4]   2>           "replica4":{
   [junit4]   2>             "core":"slice1_replica4",
   [junit4]   2>             "node_name":"baseUrl2_",
   [junit4]   2>             "base_url":"http://baseUrl2",
   [junit4]   2>             "state":"down",
   [junit4]   2>             "type":"NRT"},
   [junit4]   2>           "replica1":{
   [junit4]   2>             "core":"slice1_replica1",
   [junit4]   2>             "leader":"true",
   [junit4]   2>             "node_name":"baseUrl1_",
   [junit4]   2>             "base_url":"http://baseUrl1",
   [junit4]   2>             "state":"active",
   [junit4]   2>             "type":"NRT"},
   [junit4]   2>           "replica3":{
   [junit4]   2>             "core":"slice1_replica3",
   [junit4]   2>             "node_name":"baseUrl1_",
   [junit4]   2>             "base_url":"http://baseUrl1",
   [junit4]   2>             "state":"down",
   [junit4]   2>             "type":"NRT"},
   [junit4]   2>           "replica2":{
   [junit4]   2>             "core":"slice1_replica2",
   [junit4]   2>             "node_name":"baseUrl2_",
   [junit4]   2>             "base_url":"http://baseUrl2",
   [junit4]   2>             "state":"active",
   [junit4]   2>             "type":"NRT"}}}}},
   [junit4]   2>   "collection2":{
   [junit4]   2>     "replicationFactor":"1",
   [junit4]   2>     "maxShardsPerNode":"1",
   [junit4]   2>     "shards":{"slice1":{
   [junit4]   2>         "state":"active",
   [junit4]   2>         "replicas":{
   [junit4]   2>           "replica5":{
   [junit4]   2>             "core":"slice1_replica5",
   [junit4]   2>             "leader":"true",
   [junit4]   2>             "node_name":"baseUrl2_",
   [junit4]   2>             "base_url":"http://baseUrl2",
   [junit4]   2>             "state":"recovery_failed",
   [junit4]   2>             "type":"NRT"},
   [junit4]   2>           "replica6":{
   [junit4]   2>             "core":"slice1_replica6",
   [junit4]   2>             "node_name":"baseUrl1_",
   [junit4]   2>             "base_url":"http://baseUrl1",
   [junit4]   2>             "state":"recovering",
   [junit4]   2>             "type":"NRT"}}}}}}
   [junit4]   2> 63547 INFO  (TEST-HealthCheckHandlerTest.testFindUnhealthyCores-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Ending testFindUnhealthyCores
   [junit4]   2> 63567 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckV2Api
   [junit4]   2> 63957 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/node/health params={wt=javabin&version=2} status=0 QTime=3
   [junit4]   2> 63984 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 63984 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 63984 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
   [junit4]   2> 63985 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.Server jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 11.0.6+10
   [junit4]   2> 63988 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 63988 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 63988 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 63989 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@59ae4f82{/solr,null,AVAILABLE}
   [junit4]   2> 63993 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.AbstractConnector Started ServerConnector@461f12b5{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:53650}
   [junit4]   2> 63993 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.Server Started @64071ms
   [junit4]   2> 63993 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=53650}
   [junit4]   2> 63994 ERROR (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 63994 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 63994 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 8.6.0
   [junit4]   2> 63994 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 63994 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr
   [junit4]   2> 63994 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-05-23T04:49:27.816703Z
   [junit4]   2> 63996 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 64003 INFO  (zkConnectionManagerCallback-158-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 64003 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 64006 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 64182 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 64186 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@fc7805b[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 64186 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@fc7805b[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 64197 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@3fb61f9f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 64197 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@3fb61f9f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 64198 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:53639/solr
   [junit4]   2> 64200 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 64206 INFO  (zkConnectionManagerCallback-169-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 64206 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 64318 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 64323 INFO  (zkConnectionManagerCallback-171-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 64323 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 64341 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 64358 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.ZkController Publish node=127.0.0.1:53650_solr as DOWN
   [junit4]   2> 64362 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 64362 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53650_solr
   [junit4]   2> 64369 INFO  (zkCallback-134-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 64376 INFO  (zkCallback-170-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 64376 INFO  (zkCallback-151-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 64377 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 64378 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 64398 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 64427 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53650.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 64440 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53650.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 64441 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53650.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 64442 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [n:127.0.0.1:53650_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J4/temp/solr.handler.admin.HealthCheckHandlerTest_36D25A3BB9B293B3-001/tempDir-001/node2/.
   [junit4]   2> 64516 INFO  (qtp1214885855-178) [n:127.0.0.1:53650_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/node/health params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 64524 INFO  (zkCallback-134-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 64525 INFO  (zkCallback-151-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
   [junit4]   2> 64632 INFO  (qtp1214885855-176) [n:127.0.0.1:53650_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/node/health params={wt=javabin&version=2} status=503 QTime=0
   [junit4]   2> 64641 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=9735833
   [junit4]   2> 64642 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:53650_solr
   [junit4]   2> 64643 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.ZkController Publish this node as DOWN...
   [junit4]   2> 64643 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:53650_solr as DOWN
   [junit4]   2> 64644 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.ZkController Error publishing nodes as down. Continuing to close CoreContainer
   [junit4]   2>           => org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129)
   [junit4]   2> org.apache.solr.common.SolrException: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:129) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:834) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.Overseer.getStateUpdateQueue(Overseer.java:822) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.publishNodeAsDown(ZkController.java:2654) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkController.preClose(ZkController.java:603) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.core.CoreContainer.shutdown(CoreContainer.java:965) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.close(SolrDispatchFilter.java:337) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner.stop(JettySolrRunner.java:631) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.handler.admin.HealthCheckHandlerTest.testHealthCheckV2Api(HealthCheckHandlerTest.java:185) ~[test/:?]
   [junit4]   2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
   [junit4]   2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
   [junit4]   2> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[java/:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) ~[java/:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) ~[java/:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) ~[java/:?]
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) ~[java/:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368) ~[randomizedtesting-runner-2.7.2.jar:?]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:834) [?:?]
   [junit4]   2> Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/queue
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134) ~[zookeeper-3.5.7.jar:3.5.7]
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[zookeeper-3.5.7.jar:3.5.7]
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2021) ~[zookeeper-3.5.7.jar:3.5.7]
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$exists$3(SolrZkClient.java:315) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:315) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:108) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.ensureExists(ZkCmdExecutor.java:93) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkDistributedQueue.<init>(ZkDistributedQueue.java:127) ~[java/:?]
   [junit4]   2> 	... 46 more
   [junit4]   2> 64651 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
   [junit4]   2> 64652 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4c7da9cd: rootName = solr_53650, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@1350227a
   [junit4]   2> 64660 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
   [junit4]   2> 64660 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6569906d: rootName = solr_53650, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@132469d1
   [junit4]   2> 64663 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
   [junit4]   2> 64664 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7be9bd0e: rootName = solr_53650, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@4fbd6a5e
   [junit4]   2> 64664 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
   [junit4]   2> 64675 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@461f12b5{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:0}
   [junit4]   2> 64677 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@59ae4f82{/solr,null,UNAVAILABLE}
   [junit4]   2> 64677 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 64679 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckV2Api-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Ending testHealthCheckV2Api
   [junit4]   2> 64702 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandlerWithCloudClient-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckHandlerWithCloudClient
   [junit4]   2> 64704 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandlerWithCloudClient-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Ending testHealthCheckHandlerWithCloudClient
   [junit4]   2> 64726 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandlerSolrJ-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckHandlerSolrJ
   [junit4]   2> 64756 INFO  (qtp642402007-139) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/health params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 64758 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandlerSolrJ-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Ending testHealthCheckHandlerSolrJ
   [junit4]   2> 64781 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testHealthCheckHandler
   [junit4]   2> 64784 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/health params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 64815 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/health params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 64859 INFO  (qtp642402007-139) [n:127.0.0.1:53644_solr     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=_default&name=test&nrtReplicas=1&action=CREATE&numShards=1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 64881 INFO  (OverseerThreadFactory-142-thread-1-processing-n:127.0.0.1:53644_solr) [n:127.0.0.1:53644_solr     ] o.a.s.c.a.c.CreateCollectionCmd Create collection test
   [junit4]   2> 65062 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 65063 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CONTAINER.cores&wt=javabin&version=2&group=solr.node} status=0 QTime=10
   [junit4]   2> 65080 INFO  (qtp642402007-137) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CONTAINER.cores&wt=javabin&version=2&group=solr.node} status=0 QTime=2
   [junit4]   2> 65121 INFO  (OverseerStateUpdate-72089327819948037-127.0.0.1:53644_solr-n_0000000000) [n:127.0.0.1:53644_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"test",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"test_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:53644/solr",
   [junit4]   2>   "node_name":"127.0.0.1:53644_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 65350 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr    x:test_shard1_replica_n1 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=_default&newCollection=true&collection=test&version=2&replicaType=NRT&coreNodeName=core_node2&name=test_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin
   [junit4]   2> 65386 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.6.0
   [junit4]   2> 65421 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.s.IndexSchema Schema name=default-config
   [junit4]   2> 65740 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
   [junit4]   2> 65871 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.CoreContainer Creating SolrCore 'test_shard1_replica_n1' using configuration from configset _default, trusted=true
   [junit4]   2> 65873 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_53644.solr.core.test.shard1.replica_n1' (registry 'solr.core.test.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@427593b5
   [junit4]   2> 65877 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.SolrCore [[test_shard1_replica_n1] ] Opening new SolrCore at [/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J4/temp/solr.handler.admin.HealthCheckHandlerTest_36D25A3BB9B293B3-001/tempDir-001/node1/./test_shard1_replica_n1], dataDir=[/Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J4/temp/solr.handler.admin.HealthCheckHandlerTest_36D25A3BB9B293B3-001/tempDir-001/node1/./test_shard1_replica_n1/data/]
   [junit4]   2> 66145 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 66145 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 66147 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 
   [junit4]   2> 66147 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 66156 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/_default
   [junit4]   2> 66157 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/_default
   [junit4]   2> 66159 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.s.ZkIndexSchemaReader Creating ZooKeeper watch for the managed schema at /configs/_default/managed-schema
   [junit4]   2> 66161 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.s.ZkIndexSchemaReader Current schema version 0 is already the latest
   [junit4]   2> 66162 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.h.c.SpellCheckComponent Initializing spell checkers
   [junit4]   2> 66165 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.s.DirectSolrSpellChecker init: {name=default,field=_text_,classname=solr.DirectSolrSpellChecker,distanceMeasure=internal,accuracy=0.5,maxEdits=2,minPrefix=1,maxInspections=5,minQueryLength=4,maxQueryFrequency=0.01}
   [junit4]   2> 66172 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
   [junit4]   2> 66174 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1667455380352925696
   [junit4]   2> 66180 INFO  (searcherExecutor-186-thread-1-processing-n:127.0.0.1:53644_solr x:test_shard1_replica_n1 c:test s:shard1 r:core_node2) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.QuerySenderListener QuerySenderListener done.
   [junit4]   2> 66180 INFO  (searcherExecutor-186-thread-1-processing-n:127.0.0.1:53644_solr x:test_shard1_replica_n1 c:test s:shard1 r:core_node2) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
   [junit4]   2> 66182 INFO  (searcherExecutor-186-thread-1-processing-n:127.0.0.1:53644_solr x:test_shard1_replica_n1 c:test s:shard1 r:core_node2) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.SolrCore [test_shard1_replica_n1]  Registered new searcher autowarm time: 0 ms: Collection: 'test'
   [junit4]   2> 66200 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 66205 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/test/leaders/shard1
   [junit4]   2> 66223 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 66223 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 66224 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync replicas to https://127.0.0.1:53644/solr/test_shard1_replica_n1/
   [junit4]   2> 66226 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 66227 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.SyncStrategy https://127.0.0.1:53644/solr/test_shard1_replica_n1/ has no replicas
   [junit4]   2> 66229 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/test/leaders/shard1/leader after winning as /collections/test/leader_elect/shard1/election/72089327819948037-core_node2-n_0000000000
   [junit4]   2> 66240 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I am the new leader: https://127.0.0.1:53644/solr/test_shard1_replica_n1/ shard1
   [junit4]   2> 66356 INFO  (zkCallback-134-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66357 INFO  (zkCallback-134-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66359 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr c:test s:shard1 r:core_node2 x:test_shard1_replica_n1 ] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 66366 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=_default&newCollection=true&collection=test&version=2&replicaType=NRT&coreNodeName=core_node2&name=test_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin} status=0 QTime=1016
   [junit4]   2> 66375 INFO  (qtp642402007-139) [n:127.0.0.1:53644_solr     ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 66470 INFO  (zkCallback-134-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66470 INFO  (zkCallback-134-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66470 INFO  (zkCallback-134-thread-3) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66472 INFO  (qtp642402007-139) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=_default&name=test&nrtReplicas=1&action=CREATE&numShards=1&property.solr.directoryFactory=solr.StandardDirectoryFactory&wt=javabin&version=2} status=0 QTime=1613
   [junit4]   2> 66479 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/info/health params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 66488 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 66499 INFO  (qtp642402007-139) [n:127.0.0.1:53644_solr     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :delete with params name=test&action=DELETE&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 66508 INFO  (OverseerCollectionConfigSetProcessor-72089327819948037-127.0.0.1:53644_solr-n_0000000000) [n:127.0.0.1:53644_solr     ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
   [junit4]   2> 66510 INFO  (OverseerThreadFactory-142-thread-2-processing-n:127.0.0.1:53644_solr) [n:127.0.0.1:53644_solr     ] o.a.s.c.a.c.OverseerCollectionMessageHandler Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true&deleteMetricsHistory=true, asyncId=null
   [junit4]   2> 66513 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr    x:test_shard1_replica_n1 ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.test.shard1.replica_n1 tag=null
   [junit4]   2> 66513 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr    x:test_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3485df0f: rootName = solr_53644, domain = solr.core.test.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.test.shard1.replica_n1/com.codahale.metrics.MetricRegistry@250d7e8d
   [junit4]   2> 66535 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.c.SolrCore [test_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e0dbc3a
   [junit4]   2> 66535 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.test.shard1.replica_n1 tag=SolrCore@3e0dbc3a
   [junit4]   2> 66535 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.test.shard1.leader tag=SolrCore@3e0dbc3a
   [junit4]   2> 66535 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
   [junit4]   2> 66544 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/test/terms/shard1 to Terms{values={}, version=1}
   [junit4]   2> 66552 INFO  (qtp642402007-138) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&deleteMetricsHistory=true&core=test_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=39
   [junit4]   2> 66681 INFO  (zkCallback-134-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66681 INFO  (zkCallback-134-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66682 INFO  (zkCallback-134-thread-3) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/test/state.json] for collection [test] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 66722 INFO  (qtp642402007-139) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=test&action=DELETE&wt=javabin&version=2} status=0 QTime=223
   [junit4]   2> 66729 INFO  (qtp642402007-137) [n:127.0.0.1:53644_solr     ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :list with params action=LIST&wt=javabin&version=2 
   [junit4]   2> 66731 INFO  (qtp642402007-137) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/configs params={action=LIST&wt=javabin&version=2} status=0 QTime=1
   [junit4]   2> 66737 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr     ] o.a.s.h.a.ConfigSetsHandler Invoked ConfigSet Action :delete with params name=conf&action=DELETE&wt=javabin&version=2 
   [junit4]   2> 66746 INFO  (OverseerThreadFactory-142-thread-3-processing-n:127.0.0.1:53644_solr) [n:127.0.0.1:53644_solr     ] o.a.s.c.OverseerConfigSetMessageHandler OverseerConfigSetMessageHandler.processMessage : delete, {
   [junit4]   2>   "name":"conf",
   [junit4]   2>   "operation":"configsets:delete"}
   [junit4]   2> 66758 INFO  (OverseerCollectionConfigSetProcessor-72089327819948037-127.0.0.1:53644_solr-n_0000000000) [n:127.0.0.1:53644_solr     ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist. Requestor may have disconnected from ZooKeeper
   [junit4]   2> 66759 INFO  (OverseerThreadFactory-142-thread-3-processing-n:127.0.0.1:53644_solr) [n:127.0.0.1:53644_solr     ] o.a.s.c.OverseerConfigSetMessageHandler no existing ConfigSet properties found
   [junit4]   2> 66768 INFO  (qtp642402007-136) [n:127.0.0.1:53644_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/configs params={name=conf&action=DELETE&wt=javabin&version=2} status=0 QTime=31
   [junit4]   2> 66774 WARN  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 66774 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 66774 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
   [junit4]   2> 66774 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.Server jetty-9.4.27.v20200227; built: 2020-02-27T18:37:21.340Z; git: a304fd9f351f337e7c0e2a7c28878dd536149c6c; jvm 11.0.6+10
   [junit4]   2> 66780 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 66780 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 66780 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 66782 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@125c833c{/solr,null,AVAILABLE}
   [junit4]   2> 66786 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.AbstractConnector Started ServerConnector@3ddfdb1b{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:53664}
   [junit4]   2> 66786 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.e.j.s.Server Started @66864ms
   [junit4]   2> 66786 INFO  (TEST-HealthCheckHandlerTest.testHealthCheckHandler-seed#[36D25A3BB9B293B3]) [     ] o.a.s.c.s.e.JettySo

[...truncated too long message...]

ecutor.lambda$execute$0(ExecutorUtil.java:210)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 418522 WARN  (OverseerThreadFactory-2172-thread-3-processing-n:127.0.0.1:55926_solr) [n:127.0.0.1:55926_solr     ] o.a.s.c.OverseerTaskProcessor Resetting task: /overseer/collection-queue-work/qn-0000000003, requestid: 001, taskKey: null
   [junit4]   2> 418522 ERROR (OverseerThreadFactory-2172-thread-3-processing-n:127.0.0.1:55926_solr) [n:127.0.0.1:55926_solr     ] o.a.s.c.OverseerTaskProcessor :org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer/collection-map-running/mn-001
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
   [junit4]   2> 	at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
   [junit4]   2> 	at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.lambda$delete$1(SolrZkClient.java:264)
   [junit4]   2> 	at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71)
   [junit4]   2> 	at org.apache.solr.common.cloud.SolrZkClient.delete(SolrZkClient.java:263)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedMap.remove(DistributedMap.java:98)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.resetTaskWithException(OverseerTaskProcessor.java:600)
   [junit4]   2> 	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:566)
   [junit4]   2> 	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
   [junit4]   2> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 418523 INFO  (OverseerAutoScalingTriggerThread-72089348398972948-127.0.0.1:55926_solr-n_0000000000) [     ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
   [junit4]   2> 418524 INFO  (jetty-closer-2296-thread-2) [     ] o.a.s.c.Overseer Overseer (id=72089348398972948-127.0.0.1:55926_solr-n_0000000000) closing
   [junit4]   2> 418525 INFO  (closeThreadPool-2332-thread-1) [     ] o.a.s.c.Overseer Overseer (id=72089348398972948-127.0.0.1:55926_solr-n_0000000000) closing
   [junit4]   2> 418531 INFO  (jetty-closer-2296-thread-2) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@1bf1d966{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
   [junit4]   2> 418533 INFO  (jetty-closer-2296-thread-2) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@3995f828{/solr,null,UNAVAILABLE}
   [junit4]   2> 418536 INFO  (jetty-closer-2296-thread-2) [     ] o.e.j.s.session node0 Stopped scavenging
   [junit4]   2> 418539 INFO  (SUITE-ReplaceNodeTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
   [junit4]   2> 418747 WARN  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Watch limit violations: 
   [junit4]   2> Maximum concurrent create/delete watches above limit:
   [junit4]   2> 
   [junit4]   2> 	16	/solr/collections/replacenodetest_coll/terms/shard5
   [junit4]   2> 	16	/solr/collections/replacenodetest_coll/terms/shard1
   [junit4]   2> 	7	/solr/configs/conf1
   [junit4]   2> 	7	/solr/clusterprops.json
   [junit4]   2> 	7	/solr/aliases.json
   [junit4]   2> 	6	/solr/packages.json
   [junit4]   2> 	6	/solr/security.json
   [junit4]   2> 	6	/solr/collections/replacenodetest_coll/collectionprops.json
   [junit4]   2> 	5	/solr/collections/replacenodetest_coll/terms/shard2
   [junit4]   2> 	5	/solr/collections/replacenodetest_coll/terms/shard4
   [junit4]   2> 	3	/solr/collections/replacenodetest_coll/terms/shard3
   [junit4]   2> 
   [junit4]   2> Maximum concurrent data watches above limit:
   [junit4]   2> 
   [junit4]   2> 	314	/solr/collections/replacenodetest_coll/state.json
   [junit4]   2> 	7	/solr/clusterstate.json
   [junit4]   2> 	3	/solr/overseer_elect/election/72089348398972944-127.0.0.1:55935_solr-n_0000000002
   [junit4]   2> 	2	/solr/overseer_elect/election/72089348398972947-127.0.0.1:55933_solr-n_0000000001
   [junit4]   2> 
   [junit4]   2> Maximum concurrent children watches above limit:
   [junit4]   2> 
   [junit4]   2> 	81	/solr/overseer/queue
   [junit4]   2> 	38	/solr/live_nodes
   [junit4]   2> 	14	/solr/collections
   [junit4]   2> 	7	/solr/overseer/collection-queue-work
   [junit4]   2> 
   [junit4]   2> 418752 INFO  (SUITE-ReplaceNodeTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer waitForServerDown: 127.0.0.1:55920
   [junit4]   2> 418752 INFO  (SUITE-ReplaceNodeTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:55920
   [junit4]   2> 418752 INFO  (SUITE-ReplaceNodeTest-seed#[36D25A3BB9B293B3]-worker) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 55920
   [junit4]   2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-8.x-MacOSX/solr/build/solr-core/test/J1/temp/solr.cloud.ReplaceNodeTest_36D25A3BB9B293B3-001
   [junit4]   2> May 23, 2020 4:55:22 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene86): {}, docValues:{}, maxPointsInLeafNode=1084, maxMBSortInHeap=6.717339406620854, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@7b29e10c), locale=en-SG, timezone=Indian/Reunion
   [junit4]   2> NOTE: Mac OS X 10.14.6 x86_64/AdoptOpenJDK 11.0.6 (64-bit)/cpus=6,threads=1,free=192437600,total=405274624
   [junit4]   2> NOTE: All tests run in this JVM: [InfoHandlerTest, TestSimpleQParserPlugin, TestSolrCoreParser, Tagger2Test, TemplateUpdateProcessorTest, CoreAdminRequestStatusTest, TestCollectionAPIs, TestMacroExpander, XMLAtomicUpdateMultivalueTest, JavaBinAtomicUpdateMultivalueTest, SpellingQueryConverterTest, DistributedUpdateProcessorTest, SyncSliceTest, TestSolrCLIRunExample, EchoParamsTest, MultiThreadedOCPTest, TestConfigSetsAPIZkFailure, PeerSyncWithBufferUpdatesTest, AdminHandlersProxyTest, TimeZoneUtilsTest, TestFoldingMultitermQuery, TestDocTermOrds, RecoveryAfterSoftCommitTest, TestDirectoryFactory, DistributedFacetPivotSmallAdvancedTest, TestInfoStreamLogging, TestFieldCacheSortRandom, RegexBytesRefFilterTest, TestSolrCloudWithHadoopAuthPlugin, ReplaceNodeTest]
   [junit4] Completed [167/910 (5!)] on J1 in 44.10s, 1 test, 1 failure <<< FAILURES!

[...truncated 50325 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/1033/consoleText

[repro] Revision: 5e7be63ca7fbd234526fceb2d7a0594a54f90670

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC"
[repro] Repro line:  ant test  -Dtestcase=HealthCheckHandlerTest -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=luy-KE -Dtests.timezone=Iran -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=DistributedUpdateProcessorTest -Dtests.method=testVersionAdd -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=os-GE -Dtests.timezone=Asia/Harbin -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=DaemonStreamApiTest -Dtests.method=testAPIs -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=yav-CM -Dtests.timezone=Europe/Madrid -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=DaemonStreamApiTest -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=yav-CM -Dtests.timezone=Europe/Madrid -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=MultiSolrCloudTestCaseTest -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=th-TH-u-nu-thai-x-lvariant-TH -Dtests.timezone=Asia/Aqtobe -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=ReplaceNodeTest -Dtests.method=test -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=en-SG -Dtests.timezone=Indian/Reunion -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] JUnit rest result XML files will be moved to: ./repro-reports
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       DistributedUpdateProcessorTest
[repro]       DaemonStreamApiTest
[repro]       MultiSolrCloudTestCaseTest
[repro]       ReplaceNodeTest
[repro]       HealthCheckHandlerTest
[repro] ant compile-test

[...truncated 3486 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=25 -Dtests.class="*.DistributedUpdateProcessorTest|*.DaemonStreamApiTest|*.MultiSolrCloudTestCaseTest|*.ReplaceNodeTest|*.HealthCheckHandlerTest" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC" -Dtests.seed=36D25A3BB9B293B3 -Dtests.slow=true -Dtests.locale=os-GE -Dtests.timezone=Asia/Harbin -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 191 lines...]
[repro] Failures w/original seeds:
[repro]   0/5 failed: org.apache.solr.cloud.MultiSolrCloudTestCaseTest
[repro]   0/5 failed: org.apache.solr.cloud.ReplaceNodeTest
[repro]   0/5 failed: org.apache.solr.handler.admin.DaemonStreamApiTest
[repro]   0/5 failed: org.apache.solr.handler.admin.HealthCheckHandlerTest
[repro]   0/5 failed: org.apache.solr.update.processor.DistributedUpdateProcessorTest
[repro] Exiting with code 0

[...truncated 73 lines...]