[JENKINS] Solr » Solr-NightlyTests-main - Build # 1363 - Still Unstable!

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/1363/

8 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
java.lang.AssertionError: expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:A582D9B9AF75A5F8]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.cloud.AbstractChaosMonkeySafeLeaderTestBase.test(AbstractChaosMonkeySafeLeaderTestBase.java:166)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test

Error Message:
java.lang.AssertionError: expected:<0> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<3>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:A582D9B9AF75A5F8]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.cloud.AbstractChaosMonkeySafeLeaderTestBase.test(AbstractChaosMonkeySafeLeaderTestBase.java:166)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test.classMethod

Error Message:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotNull(Assert.java:756)
	at org.junit.Assert.assertNull(Assert.java:738)
	at org.apache.solr.SolrTestCase$1.afterIfSuccessful(SolrTestCase.java:103)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test

Error Message:
java.lang.AssertionError: expected:<0> but was:<12>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<12>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:A582D9B9AF75A5F8]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.cloud.AbstractChaosMonkeySafeLeaderTestBase.test(AbstractChaosMonkeySafeLeaderTestBase.java:166)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:2E573B0BDF97A22]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:2E573B0BDF97A22]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:2E573B0BDF97A22]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([2DD6E6630189C800:2E573B0BDF97A22]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)




Build Log:
[...truncated 50375 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest&tests.test=test http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsChaosMonkeySafeLeaderTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=2DD6E6630189C800 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=UTF-8 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest&tests.test=testDataDirAndUlogAreMaintained http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=2DD6E6630189C800 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=UTF-8 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=2DD6E6630189C800 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=UTF-8 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.ChaosMonkeySafeLeaderTest&tests.test=test http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.cloud.ChaosMonkeySafeLeaderTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=2DD6E6630189C800 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=UTF-8 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1h 59m 25s
152 actionable tasks: 152 executed

Publishing build scan...

The Develocity server (develocity.apache.org) rejected the access key with prefix 'eyJraWQiOiJhdXRoLWtleS1pZC1mOThkZjViNi01NWYwLTQ3MmEtYjY3MS1mNTc2NzcyOTRiMWEiLCJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc19hbm9ueW1vdXMiOmZhbHNlLCJwZXJtaXNzaW9ucyI6WyJWSUVXX1NDQU4iLCJFWFBPUlRfREFUQSIsIlBVQkxJU0hfU0NBTiIsIlBSRURJQ1RJVkVfVEVTVF9TRUxFQ1RJT04iLCJSRUFEX0NBQ0hFIiwiQUNDRVNTX0FMTF9EQVRBIiwiQUNDRVNTX0RBVEFfV0lUSE9VVF9BU1NPQ0lBVEVEX1BST0pFQ1QiLCJNWV9TRVRUSU5HUyIsIlJFQURfVkVSU0lPTiJdLCJhbGxEYXRhUGVybWlzc2lvbnMiOlsiVklFV19TQ0FOIiwiUFJFRElDVElWRV9URVNUX1NFTEVDVElPTiIsIlJFQURfQ0FDSEUiLCJBQ0NFU1NfQUxMX0RBVEEiLCJBQ0NFU1NfREFUQV9XSVRIT1VUX0FTU09DSUFURURfUFJPSkVDVCIsIkVYUE9SVF9EQVRBIiwiUFVCTElTSF9TQ0FOIiwiTVlfU0VUVElOR1MiLCJSRUFEX1ZFUlNJT04iXSwicHJvamVjdHMiOnt9LCJ'.
Your access key has expired - please visit https://develocity.apache.org/settings/access-keys to refresh it.

Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
‘**/*.events’ doesn’t match anything: ‘**’ exists but not ‘**/*.events’
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
[Checks API] No suitable checks publisher found.
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Solr » Solr-NightlyTests-main - Build # 1366 - Still Unstable!

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/1366/

18 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:292ACD46EEB9290D]:0)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1132)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:2009)
	at app//org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:281)
	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:292ACD46EEB9290D]:0)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1132)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:2009)
	at app//org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:281)
	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:292ACD46EEB9290D]:0)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1132)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:2009)
	at app//org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:281)
	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:292ACD46EEB9290D]:0)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1132)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:2009)
	at app//org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:281)
	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.cloud.LeaderElectionTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest: 
   1) Thread[id=55068, name=TEST-LeaderElectionTest.testStressElection-seed#[A17EF29C404544F5]-SendThread(127.0.0.1:44471), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.base/java.lang.Thread.sleep0(Native Method)
        at java.base/java.lang.Thread.sleep(Thread.java:509)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:362)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1189)
   2) Thread[id=55069, name=TEST-LeaderElectionTest.testStressElection-seed#[A17EF29C404544F5]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at java.base/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
        at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
        at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
        at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:535)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 2 threads leaked from SUITE scope at org.apache.solr.cloud.LeaderElectionTest: 
   1) Thread[id=55068, name=TEST-LeaderElectionTest.testStressElection-seed#[A17EF29C404544F5]-SendThread(127.0.0.1:44471), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.base/java.lang.Thread.sleep0(Native Method)
        at java.base/java.lang.Thread.sleep(Thread.java:509)
        at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:362)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1189)
   2) Thread[id=55069, name=TEST-LeaderElectionTest.testStressElection-seed#[A17EF29C404544F5]-EventThread, state=WAITING, group=TGRP-LeaderElectionTest]
        at java.base/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base/java.util.concurrent.locks.LockSupport.park(LockSupport.java:371)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionNode.block(AbstractQueuedSynchronizer.java:519)
        at java.base/java.util.concurrent.ForkJoinPool.unmanagedBlock(ForkJoinPool.java:3780)
        at java.base/java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3725)
        at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1707)
        at java.base/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:435)
        at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:535)
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5]:0)


FAILED:  org.apache.solr.cloud.LeaderElectionTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=55068, name=TEST-LeaderElectionTest.testStressElection-seed#[A17EF29C404544F5]-SendThread(127.0.0.1:44471), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.base/java.lang.Thread.sleep0(Native Method)
        at java.base/java.lang.Thread.sleep(Thread.java:509)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1129)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=55068, name=TEST-LeaderElectionTest.testStressElection-seed#[A17EF29C404544F5]-SendThread(127.0.0.1:44471), state=TIMED_WAITING, group=TGRP-LeaderElectionTest]
        at java.base/java.lang.Thread.sleep0(Native Method)
        at java.base/java.lang.Thread.sleep(Thread.java:509)
        at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1129)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1192)
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5]:0)


FAILED:  org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit2

Error Message:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=80878, name=zkCallback-52564-thread-1, state=RUNNABLE, group=TGRP-ShardSplitTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=80878, name=zkCallback-52564-thread-1, state=RUNNABLE, group=TGRP-ShardSplitTest]
Caused by: org.apache.solr.common.SolrException: core not found: collection1_shard1_0_replica_n9
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5]:0)
	at app//org.apache.solr.handler.admin.PrepRecoveryOp.lambda$execute$0(PrepRecoveryOp.java:94)
	at app//org.apache.solr.common.cloud.ZkStateReader.lambda$waitForState$10(ZkStateReader.java:1626)
	at app//org.apache.solr.common.cloud.ZkStateReader$DocCollectionAndLiveNodesWatcherWrapper.onChange(ZkStateReader.java:2129)
	at app//org.apache.solr.common.cloud.ZkStateReader.lambda$refreshLiveNodes$2(ZkStateReader.java:832)
	at java.base@21.0.4/java.util.concurrent.ConcurrentHashMap$KeySetView.forEach(ConcurrentHashMap.java:4709)
	at app//org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:830)
	at app//org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.refreshAndWatch(ZkStateReader.java:1367)
	at app//org.apache.solr.common.cloud.ZkStateReader$LiveNodeWatcher.process(ZkStateReader.java:1362)
	at app//org.apache.solr.common.cloud.SolrZkClient$ProcessWatchWithExecutor.lambda$process$0(SolrZkClient.java:1179)
	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base@21.0.4/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.search.TestThinCache.testSimple

Error Message:
java.lang.AssertionError: expected:<0> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<0> but was:<1>
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:99CDD66267B69024]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:120)
	at org.junit.Assert.assertEquals(Assert.java:146)
	at org.apache.solr.search.TestThinCache.testSimple(TestThinCache.java:139)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test.classMethod

Error Message:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms, ZkShardTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 4 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms, ZkShardTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms, ZkShardTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 4 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms, ZkShardTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotNull(Assert.java:756)
	at org.junit.Assert.assertNull(Assert.java:738)
	at org.apache.solr.SolrTestCase$1.afterIfSuccessful(SolrTestCase.java:103)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:292ACD46EEB9290D]:0)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1132)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:2009)
	at app//org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:247)
	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.4/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.

Stack Trace:
org.apache.solr.common.SolrException: Could not find a healthy node to handle the request.
	at __randomizedtesting.SeedInfo.seed([A17EF29C404544F5:292ACD46EEB9290D]:0)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1132)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:970)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.commit(AbstractFullDistribZkTestBase.java:2009)
	at app//org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:247)
	at java.base@21.0.4/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.4/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSing

[...truncated too long message...]

.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest&tests.test=testDataDirAndUlogAreMaintained http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsUnloadDistributedZkTest.test (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsUnloadDistributedZkTest&tests.test=test http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.HdfsUnloadDistributedZkTest.test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsUnloadDistributedZkTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsUnloadDistributedZkTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest&tests.test=test http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.LeaderElectionTest (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.LeaderElectionTest
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.LeaderElectionTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.cloud.LeaderElectionTest" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.search.TestThinCache.testSimple (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.search.TestThinCache&tests.test=testSimple http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.search.TestThinCache.testSimple
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestThinCache.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.search.TestThinCache.testSimple" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit2 (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.api.collections.ShardSplitTest&tests.test=testSplitAfterFailedSplit2 http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit2
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.api.collections.ShardSplitTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.cloud.api.collections.ShardSplitTest.testSplitAfterFailedSplit2" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A17EF29C404544F5 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1h 58m 51s
152 actionable tasks: 152 executed

Publishing build scan...

The Develocity server (develocity.apache.org) rejected the access key with prefix 'eyJraWQiOiJhdXRoLWtleS1pZC1mOThkZjViNi01NWYwLTQ3MmEtYjY3MS1mNTc2NzcyOTRiMWEiLCJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc19hbm9ueW1vdXMiOmZhbHNlLCJwZXJtaXNzaW9ucyI6WyJWSUVXX1NDQU4iLCJFWFBPUlRfREFUQSIsIlBVQkxJU0hfU0NBTiIsIlBSRURJQ1RJVkVfVEVTVF9TRUxFQ1RJT04iLCJSRUFEX0NBQ0hFIiwiQUNDRVNTX0FMTF9EQVRBIiwiQUNDRVNTX0RBVEFfV0lUSE9VVF9BU1NPQ0lBVEVEX1BST0pFQ1QiLCJNWV9TRVRUSU5HUyIsIlJFQURfVkVSU0lPTiJdLCJhbGxEYXRhUGVybWlzc2lvbnMiOlsiVklFV19TQ0FOIiwiUFJFRElDVElWRV9URVNUX1NFTEVDVElPTiIsIlJFQURfQ0FDSEUiLCJBQ0NFU1NfQUxMX0RBVEEiLCJBQ0NFU1NfREFUQV9XSVRIT1VUX0FTU09DSUFURURfUFJPSkVDVCIsIkVYUE9SVF9EQVRBIiwiUFVCTElTSF9TQ0FOIiwiTVlfU0VUVElOR1MiLCJSRUFEX1ZFUlNJT04iXSwicHJvamVjdHMiOnt9LCJ'.
Your access key has expired - please visit https://develocity.apache.org/settings/access-keys to refresh it.

Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
‘**/*.events’ doesn’t match anything: ‘**’ exists but not ‘**/*.events’
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
[Checks API] No suitable checks publisher found.
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Solr » Solr-NightlyTests-main - Build # 1365 - Still Unstable!

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/1365/

8 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test

Error Message:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: http://127.0.0.1:45579/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: http://127.0.0.1:45579/solr
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:494F45ED0F22983B]:0)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:720)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
	at app//org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:520)
	at app//org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:502)
	at app//org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:529)
	at app//org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:468)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1139)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:834)
	at app//org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:766)
	at app//org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1195)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:2092)
	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:2152)
	at app//org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test(ChaosMonkeyNothingIsSafeWithPullReplicasTest.java:341)
	at java.base@21.0.3/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.3/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583)
Caused by: java.net.SocketTimeoutException: Read timed out
	at java.base/sun.nio.ch.NioSocketImpl.timedRead(NioSocketImpl.java:278)
	at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:304)
	at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:346)
	at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:796)
	at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1099)
	at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
	at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
	at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:280)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
	at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
	at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
	at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
	at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:157)
	at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
	at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:567)
	... 59 more


FAILED:  org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test.classMethod

Error Message:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [ZkShardTerms, ZkShardTerms, ZkCollectionTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 4 object(s) that were not released!!! [ZkShardTerms, ZkShardTerms, ZkCollectionTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not released!!! [ZkShardTerms, ZkShardTerms, ZkCollectionTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 4 object(s) that were not released!!! [ZkShardTerms, ZkShardTerms, ZkCollectionTerms, ZkCollectionTerms]
org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotNull(Assert.java:756)
	at org.junit.Assert.assertNull(Assert.java:738)
	at org.apache.solr.SolrTestCase$1.afterIfSuccessful(SolrTestCase.java:103)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test

Error Message:
java.lang.AssertionError: There are still nodes recovering - waited for 20SECONDS

Stack Trace:
java.lang.AssertionError: There are still nodes recovering - waited for 20SECONDS
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:494F45ED0F22983B]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:223)
	at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:156)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:1169)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForThingsToLevelOut(AbstractFullDistribZkTestBase.java:1884)
	at org.apache.solr.cloud.AbstractChaosMonkeyNothingIsSafeTestBase.test(AbstractChaosMonkeyNothingIsSafeTestBase.java:269)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1070)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1041)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest.testSocketTimeoutOnCommit

Error Message:
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://127.0.0.1:46425

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occurred when talking to server at: http://127.0.0.1:46425
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:DDFB2CBFA38A3B47]:0)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:723)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:229)
	at app//org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient.request(ConcurrentUpdateSolrClient.java:508)
	at app//org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:253)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:467)
	at app//org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:443)
	at app//org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest.testSocketTimeoutOnCommit(ConcurrentUpdateSolrClientBuilderTest.java:68)
	at java.base@21.0.3/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base@21.0.3/java.lang.reflect.Method.invoke(Method.java:580)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base@21.0.3/java.lang.Thread.run(Thread.java:1583)
Caused by: org.apache.http.conn.UnsupportedSchemeException: http protocol is not supported
	at app//org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:109)
	at app//org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at app//org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at app//org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at app//org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at app//org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at app//org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at app//org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at app//org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
	at app//org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:567)
	... 23 more


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:EE28EFE41DAE47E1]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:EE28EFE41DAE47E1]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:EE28EFE41DAE47E1]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([C11B7A37A1DEF5C3:EE28EFE41DAE47E1]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)




Build Log:
[...truncated 262805 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest.testSocketTimeoutOnCommit (:solr:solrj)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest&tests.test=testSocketTimeoutOnCommit http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest.testSocketTimeoutOnCommit
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest.txt
    Reproduce with: ./gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClientBuilderTest.testSocketTimeoutOnCommit" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=C11B7A37A1DEF5C3 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest&tests.test=test http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsChaosMonkeyNothingIsSafeTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=C11B7A37A1DEF5C3 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.HdfsBasicDistributedZk2Test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=C11B7A37A1DEF5C3 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest&tests.test=testDataDirAndUlogAreMaintained http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=C11B7A37A1DEF5C3 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest&tests.test=test http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest.test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=C11B7A37A1DEF5C3 -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=ISO-8859-1 -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1h 58m 58s
152 actionable tasks: 152 executed

Publishing build scan...

The Develocity server (develocity.apache.org) rejected the access key with prefix 'eyJraWQiOiJhdXRoLWtleS1pZC1mOThkZjViNi01NWYwLTQ3MmEtYjY3MS1mNTc2NzcyOTRiMWEiLCJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc19hbm9ueW1vdXMiOmZhbHNlLCJwZXJtaXNzaW9ucyI6WyJWSUVXX1NDQU4iLCJFWFBPUlRfREFUQSIsIlBVQkxJU0hfU0NBTiIsIlBSRURJQ1RJVkVfVEVTVF9TRUxFQ1RJT04iLCJSRUFEX0NBQ0hFIiwiQUNDRVNTX0FMTF9EQVRBIiwiQUNDRVNTX0RBVEFfV0lUSE9VVF9BU1NPQ0lBVEVEX1BST0pFQ1QiLCJNWV9TRVRUSU5HUyIsIlJFQURfVkVSU0lPTiJdLCJhbGxEYXRhUGVybWlzc2lvbnMiOlsiVklFV19TQ0FOIiwiUFJFRElDVElWRV9URVNUX1NFTEVDVElPTiIsIlJFQURfQ0FDSEUiLCJBQ0NFU1NfQUxMX0RBVEEiLCJBQ0NFU1NfREFUQV9XSVRIT1VUX0FTU09DSUFURURfUFJPSkVDVCIsIkVYUE9SVF9EQVRBIiwiUFVCTElTSF9TQ0FOIiwiTVlfU0VUVElOR1MiLCJSRUFEX1ZFUlNJT04iXSwicHJvamVjdHMiOnt9LCJ'.
Your access key has expired - please visit https://develocity.apache.org/settings/access-keys to refresh it.

Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
‘**/*.events’ doesn’t match anything: ‘**’ exists but not ‘**/*.events’
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
[Checks API] No suitable checks publisher found.
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Solr » Solr-NightlyTests-main - Build # 1364 - Still Unstable!

[Apache Jenkins Server <je...@builds.apache.org>]

Build: https://ci-builds.apache.org/job/Solr/job/Solr-NightlyTests-main/1364/

6 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.classMethod

Error Message:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>

Stack Trace:
java.lang.AssertionError: ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

 expected null, but was:<ObjectTracker found 2 object(s) that were not released!!! [ZkCollectionTerms, ZkShardTerms]
org.apache.solr.cloud.ZkCollectionTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkCollectionTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkCollectionTerms.<init>(ZkCollectionTerms.java:36)
	at org.apache.solr.cloud.ZkController.getCollectionTerms(ZkController.java:1799)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

org.apache.solr.cloud.ZkShardTerms:org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: org.apache.solr.cloud.ZkShardTerms
	at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:54)
	at org.apache.solr.cloud.ZkShardTerms.<init>(ZkShardTerms.java:107)
	at org.apache.solr.cloud.ZkCollectionTerms.getShard(ZkCollectionTerms.java:42)
	at org.apache.solr.cloud.ZkCollectionTerms.register(ZkCollectionTerms.java:49)
	at org.apache.solr.cloud.ZkController.register(ZkController.java:1279)
	at org.apache.solr.cloud.ZkController$RegisterCoreAsync.call(ZkController.java:273)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:380)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:145)
	at org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:487)
	at org.apache.solr.common.cloud.OnReconnect.stateChanged(OnReconnect.java:36)
	at org.apache.curator.framework.state.ConnectionStateManager.lambda$processEvents$0(ConnectionStateManager.java:256)
	at org.apache.curator.framework.listen.MappingListenerManager.lambda$forEach$0(MappingListenerManager.java:82)
	... 4 more
Caused by: java.lang.Exception: Submitter stack trace
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.execute(ExecutorUtil.java:347)
	at org.apache.curator.framework.listen.MappingListenerManager.forEach(MappingListenerManager.java:80)
	at org.apache.curator.framework.listen.StandardListenerManager.forEach(StandardListenerManager.java:81)
	at org.apache.curator.framework.state.ConnectionStateManager.processEvents(ConnectionStateManager.java:256)
	at org.apache.curator.framework.state.ConnectionStateManager.access$000(ConnectionStateManager.java:43)
	at org.apache.curator.framework.state.ConnectionStateManager$1.call(ConnectionStateManager.java:130)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	... 3 more

>
	at __randomizedtesting.SeedInfo.seed([50D25EE6EF24047C]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotNull(Assert.java:756)
	at org.junit.Assert.assertNull(Assert.java:738)
	at org.apache.solr.SolrTestCase$1.afterIfSuccessful(SolrTestCase.java:103)
	at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterIfSuccessful(TestRuleAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:37)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple

Error Message:
java.lang.AssertionError: expected:<3> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<3> but was:<2>
	at __randomizedtesting.SeedInfo.seed([50D25EE6EF24047C:68617A18C8D7D0AD]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple(TestLBHttpSolrClient.java:170)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([50D25EE6EF24047C:7FE1CB355354B65E]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([50D25EE6EF24047C:7FE1CB355354B65E]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([50D25EE6EF24047C:7FE1CB355354B65E]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)


FAILED:  org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained

Error Message:
java.lang.AssertionError: expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
	at __randomizedtesting.SeedInfo.seed([50D25EE6EF24047C:7FE1CB355354B65E]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained(MoveReplicaHdfsFailoverTest.java:113)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:1583)




Build Log:
[...truncated 8742 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple (:solr:solrj)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.client.solrj.TestLBHttpSolrClient&tests.test=testSimple http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttpSolrClient.txt
    Reproduce with: ./gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=50D25EE6EF24047C -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=US-ASCII -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained (:solr:modules:hdfs)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest&tests.test=testDataDirAndUlogAreMaintained http://fucit.org/solr-jenkins-reports/history-trend-of-recent-failures.html#series/org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/modules/hdfs/build/test-results/test/outputs/OUTPUT-org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.txt
    Reproduce with: ./gradlew :solr:modules:hdfs:test --tests "org.apache.solr.hdfs.cloud.MoveReplicaHdfsFailoverTest.testDataDirAndUlogAreMaintained" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=50D25EE6EF24047C -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=US-ASCII -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt

  - org.apache.solr.cloud.BasicDistributedZk2Test (:solr:core)
    Test history: https://develocity.apache.org/scans/tests?search.rootProjectNames=solr-root&tests.container=org.apache.solr.cloud.BasicDistributedZk2Test
    Test output: /home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.BasicDistributedZk2Test.txt
    Reproduce with: ./gradlew :solr:core:test --tests "org.apache.solr.cloud.BasicDistributedZk2Test" -Ptests.haltonfailure=false "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=50D25EE6EF24047C -Ptests.multiplier=2 -Ptests.nightly=true -Ptests.badapples=false -Ptests.file.encoding=US-ASCII -Ptests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Solr/Solr-NightlyTests-main/test-data/enwiki.random.lines.txt


Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 1h 51m 9s
152 actionable tasks: 152 executed

Publishing build scan...
https://develocity.apache.org/s/4gorpjpbahku4

Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
‘**/*.events’ doesn’t match anything: ‘**’ exists but not ‘**/*.events’
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/hs_err_pid*". Configuration error?
Recording test results
[Checks API] No suitable checks publisher found.
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)