You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@solr.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2023/06/26 22:40:37 UTC

[JENKINS] Solr » Solr-Check-9.x - Build # 4719 - Still Failing!

Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4719/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22636ms
ok 43 post solr log into solr in 8885ms
ok 44 start solr with ssl in 20255ms
ok 45 start solr with ssl and auth in 22833ms
ok 46 SOLR-11740 check 'solr stop' connection in 28610ms
ok 47 status detects locally running solr in 14662ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6520ms
ok 51 upconfig in 2628ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m23s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4893 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4893/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4892 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4892/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4891 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4891/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4890 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4890/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4889 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4889/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4888 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4888/

3 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess

Error Message:
java.lang.AssertionError: expected:<4> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<4> but was:<5>
	at __randomizedtesting.SeedInfo.seed([B64678882F9587A3:795B47894E107AF0]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess(TestCoordinatorRole.java:573)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=857, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=863, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=858, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=860, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=871, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=868, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=862, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=866, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=869, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=867, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=857, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=863, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=858, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=860, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=871, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=868, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=862, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=866, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=869, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=867, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([B64678882F9587A3]:0)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=857, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=863, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=858, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=868, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=862, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=866, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=869, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=860, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=871, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=867, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=857, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=863, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=858, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=868, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=862, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=866, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=869, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=860, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=871, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=867, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([B64678882F9587A3]:0)




Build Log:
[...truncated 9570 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22422ms
ok 43 post solr log into solr in 9105ms
ok 44 start solr with ssl in 20067ms
ok 45 start solr with ssl and auth in 23143ms
ok 46 SOLR-11740 check 'solr stop' connection in 28099ms
ok 47 status detects locally running solr in 14680ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2156ms
ok 50 copying files around in 6518ms
ok 51 upconfig in 2670ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B64678882F9587A3 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B64678882F9587A3 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B64678882F9587A3 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m21s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4887 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4887/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22377ms
ok 43 post solr log into solr in 9064ms
ok 44 start solr with ssl in 20018ms
ok 45 start solr with ssl and auth in 22713ms
ok 46 SOLR-11740 check 'solr stop' connection in 28457ms
ok 47 status detects locally running solr in 14412ms
ok 48 status does not expose cli parameters to end user in 219ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6596ms
ok 51 upconfig in 2653ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 5s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4886 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4886/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22291ms
ok 43 post solr log into solr in 9165ms
ok 44 start solr with ssl in 20389ms
ok 45 start solr with ssl and auth in 22394ms
ok 46 SOLR-11740 check 'solr stop' connection in 28104ms
ok 47 status detects locally running solr in 14342ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2164ms
ok 50 copying files around in 6472ms
ok 51 upconfig in 2615ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 55s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m15.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4885 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4885/

All tests passed

Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22870ms
ok 43 post solr log into solr in 9097ms
ok 44 start solr with ssl in 20322ms
ok 45 start solr with ssl and auth in 22865ms
ok 46 SOLR-11740 check 'solr stop' connection in 28036ms
ok 47 status detects locally running solr in 14695ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2156ms
ok 50 copying files around in 6502ms
ok 51 upconfig in 2622ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m14.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4884 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4884/

All tests passed

Build Log:
[...truncated 1252 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22481ms
ok 43 post solr log into solr in 9303ms
ok 44 start solr with ssl in 20419ms
ok 45 start solr with ssl and auth in 22697ms
ok 46 SOLR-11740 check 'solr stop' connection in 28525ms
ok 47 status detects locally running solr in 14463ms
ok 48 status does not expose cli parameters to end user in 222ms
ok 49 listing out files in 2160ms
ok 50 copying files around in 6608ms
ok 51 upconfig in 2662ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 3s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m24.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4883 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4883/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22537ms
ok 43 post solr log into solr in 9145ms
ok 44 start solr with ssl in 19963ms
ok 45 start solr with ssl and auth in 22181ms
ok 46 SOLR-11740 check 'solr stop' connection in 28065ms
ok 47 status detects locally running solr in 14349ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2166ms
ok 50 copying files around in 6520ms
ok 51 upconfig in 2627ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m20s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4882 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4882/

3 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess

Error Message:
java.lang.AssertionError: expected:<4> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<4> but was:<5>
	at __randomizedtesting.SeedInfo.seed([70558339671E1984:BF48BC38069BE4D7]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess(TestCoordinatorRole.java:573)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=424, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=411, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=412, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=408, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=410, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=409, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=413, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=418, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=424, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=411, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=412, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=408, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=410, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=409, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=413, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=418, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([70558339671E1984]:0)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=408, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=410, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=424, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=409, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=411, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=412, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=413, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=418, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=408, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=410, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=424, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=409, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=411, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=412, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=413, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=418, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([70558339671E1984]:0)




Build Log:
[...truncated 9420 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22295ms
ok 43 post solr log into solr in 9011ms
ok 44 start solr with ssl in 20115ms
ok 45 start solr with ssl and auth in 21988ms
ok 46 SOLR-11740 check 'solr stop' connection in 28321ms
ok 47 status detects locally running solr in 14357ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2146ms
ok 50 copying files around in 6447ms
ok 51 upconfig in 2600ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=70558339671E1984 -Ptests.file.encoding=UTF-8

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=70558339671E1984 -Ptests.file.encoding=UTF-8

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=70558339671E1984 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 57s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m19s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4881 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4881/

No tests ran.

Build Log:
[...truncated 119 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4880 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4880/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4879 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4879/

No tests ran.

Build Log:
[...truncated 120 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4878 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4878/

No tests ran.

Build Log:
[...truncated 132 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4877 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4877/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22629ms
ok 43 post solr log into solr in 8871ms
ok 44 start solr with ssl in 20197ms
ok 45 start solr with ssl and auth in 22199ms
ok 46 SOLR-11740 check 'solr stop' connection in 28066ms
ok 47 status detects locally running solr in 14350ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2197ms
ok 50 copying files around in 6547ms
ok 51 upconfig in 2618ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m21.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4876 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4876/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22648ms
ok 43 post solr log into solr in 9115ms
ok 44 start solr with ssl in 20505ms
ok 45 start solr with ssl and auth in 22734ms
ok 46 SOLR-11740 check 'solr stop' connection in 28397ms
ok 47 status detects locally running solr in 14782ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2199ms
ok 50 copying files around in 6623ms
ok 51 upconfig in 2634ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m27s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4875 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4875/

3 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess

Error Message:
java.lang.AssertionError: expected:<4> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<4> but was:<5>
	at __randomizedtesting.SeedInfo.seed([688489B074D448B7:A799B6B11551B5E4]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess(TestCoordinatorRole.java:573)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=851, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=850, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=853, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=869, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=868, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=859, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=852, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=857, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=849, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=865, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=851, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=850, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=853, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=869, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=868, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=859, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=852, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=857, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=849, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=865, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([688489B074D448B7]:0)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=859, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=851, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=850, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=852, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=857, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=853, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=869, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=868, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=849, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=865, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=859, name=TestCoordinatorRole-611-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=851, name=TestCoordinatorRole-611-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=850, name=TestCoordinatorRole-611-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=852, name=TestCoordinatorRole-611-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=857, name=TestCoordinatorRole-611-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=853, name=TestCoordinatorRole-611-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=869, name=TestCoordinatorRole-611-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=868, name=TestCoordinatorRole-611-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=849, name=TestCoordinatorRole-611-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=865, name=TestCoordinatorRole-611-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([688489B074D448B7]:0)




Build Log:
[...truncated 9607 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22484ms
ok 43 post solr log into solr in 8874ms
ok 44 start solr with ssl in 19961ms
ok 45 start solr with ssl and auth in 22314ms
ok 46 SOLR-11740 check 'solr stop' connection in 28298ms
ok 47 status detects locally running solr in 14405ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2150ms
ok 50 copying files around in 6514ms
ok 51 upconfig in 2623ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=688489B074D448B7 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=688489B074D448B7 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=688489B074D448B7 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 50s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m17s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4874 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4874/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22322ms
ok 43 post solr log into solr in 9047ms
ok 44 start solr with ssl in 19724ms
ok 45 start solr with ssl and auth in 22827ms
ok 46 SOLR-11740 check 'solr stop' connection in 28407ms
ok 47 status detects locally running solr in 14402ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2180ms
ok 50 copying files around in 6513ms
ok 51 upconfig in 2628ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 51s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4873 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4873/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22593ms
ok 43 post solr log into solr in 9092ms
ok 44 start solr with ssl in 20671ms
ok 45 start solr with ssl and auth in 22725ms
ok 46 SOLR-11740 check 'solr stop' connection in 28358ms
ok 47 status detects locally running solr in 14321ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2144ms
ok 50 copying files around in 6438ms
ok 51 upconfig in 2682ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 3s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m38s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4872 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4872/

3 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess

Error Message:
java.lang.AssertionError: expected:<4> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<4> but was:<5>
	at __randomizedtesting.SeedInfo.seed([6295B28CFD410321:AD888D8D9CC4FE72]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess(TestCoordinatorRole.java:573)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=690, name=TestCoordinatorRole-498-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=689, name=TestCoordinatorRole-498-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=677, name=TestCoordinatorRole-498-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=687, name=TestCoordinatorRole-498-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=685, name=TestCoordinatorRole-498-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=680, name=TestCoordinatorRole-498-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=682, name=TestCoordinatorRole-498-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=678, name=TestCoordinatorRole-498-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=684, name=TestCoordinatorRole-498-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=688, name=TestCoordinatorRole-498-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=690, name=TestCoordinatorRole-498-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=689, name=TestCoordinatorRole-498-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=677, name=TestCoordinatorRole-498-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=687, name=TestCoordinatorRole-498-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=685, name=TestCoordinatorRole-498-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=680, name=TestCoordinatorRole-498-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=682, name=TestCoordinatorRole-498-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=678, name=TestCoordinatorRole-498-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=684, name=TestCoordinatorRole-498-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=688, name=TestCoordinatorRole-498-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([6295B28CFD410321]:0)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=685, name=TestCoordinatorRole-498-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=680, name=TestCoordinatorRole-498-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=690, name=TestCoordinatorRole-498-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=682, name=TestCoordinatorRole-498-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=678, name=TestCoordinatorRole-498-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=684, name=TestCoordinatorRole-498-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=689, name=TestCoordinatorRole-498-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=677, name=TestCoordinatorRole-498-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=687, name=TestCoordinatorRole-498-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=688, name=TestCoordinatorRole-498-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=685, name=TestCoordinatorRole-498-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=680, name=TestCoordinatorRole-498-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=690, name=TestCoordinatorRole-498-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=682, name=TestCoordinatorRole-498-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=678, name=TestCoordinatorRole-498-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=684, name=TestCoordinatorRole-498-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=689, name=TestCoordinatorRole-498-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=677, name=TestCoordinatorRole-498-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=687, name=TestCoordinatorRole-498-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=688, name=TestCoordinatorRole-498-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([6295B28CFD410321]:0)




Build Log:
[...truncated 9361 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22529ms
ok 43 post solr log into solr in 8805ms
ok 44 start solr with ssl in 19707ms
ok 45 start solr with ssl and auth in 22769ms
ok 46 SOLR-11740 check 'solr stop' connection in 28081ms
ok 47 status detects locally running solr in 14386ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2154ms
ok 50 copying files around in 6487ms
ok 51 upconfig in 2606ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=6295B28CFD410321 -Ptests.file.encoding=US-ASCII

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=6295B28CFD410321 -Ptests.file.encoding=US-ASCII

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=6295B28CFD410321 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 6s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m32.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4871 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4871/

3 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess

Error Message:
java.lang.AssertionError: expected:<4> but was:<5>

Stack Trace:
java.lang.AssertionError: expected:<4> but was:<5>
	at __randomizedtesting.SeedInfo.seed([12E7BEE9D85384C5:DDFA81E8B9D67996]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.failNotEquals(Assert.java:835)
	at org.junit.Assert.assertEquals(Assert.java:647)
	at org.junit.Assert.assertEquals(Assert.java:633)
	at org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess(TestCoordinatorRole.java:573)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=406, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=407, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=409, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=422, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=421, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=413, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=408, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=418, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 10 threads leaked from SUITE scope at org.apache.solr.search.TestCoordinatorRole: 
   1) Thread[id=406, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=407, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=409, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=422, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=421, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=413, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=408, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=418, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([12E7BEE9D85384C5]:0)


FAILED:  org.apache.solr.search.TestCoordinatorRole.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=422, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=406, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=421, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=407, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=413, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=408, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=418, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=409, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=422, name=TestCoordinatorRole-362-thread-8, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=406, name=TestCoordinatorRole-362-thread-1, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=425, name=TestCoordinatorRole-362-thread-9, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   4) Thread[id=421, name=TestCoordinatorRole-362-thread-7, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=407, name=TestCoordinatorRole-362-thread-2, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=413, name=TestCoordinatorRole-362-thread-5, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   7) Thread[id=408, name=TestCoordinatorRole-362-thread-3, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   8) Thread[id=418, name=TestCoordinatorRole-362-thread-6, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=426, name=TestCoordinatorRole-362-thread-10, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=409, name=TestCoordinatorRole-362-thread-4, state=WAITING, group=TGRP-TestCoordinatorRole]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([12E7BEE9D85384C5]:0)




Build Log:
[...truncated 9534 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22590ms
ok 43 post solr log into solr in 8952ms
ok 44 start solr with ssl in 19724ms
ok 45 start solr with ssl and auth in 22435ms
ok 46 SOLR-11740 check 'solr stop' connection in 28139ms
ok 47 status detects locally running solr in 14361ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2194ms
ok 50 copying files around in 6568ms
ok 51 upconfig in 2668ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testConcurrentAccess" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=12E7BEE9D85384C5 -Ptests.file.encoding=UTF-8

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=12E7BEE9D85384C5 -Ptests.file.encoding=UTF-8

  - org.apache.solr.search.TestCoordinatorRole.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=12E7BEE9D85384C5 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 4s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m31s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4870 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4870/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22466ms
ok 43 post solr log into solr in 9110ms
ok 44 start solr with ssl in 19862ms
ok 45 start solr with ssl and auth in 22610ms
ok 46 SOLR-11740 check 'solr stop' connection in 28802ms
ok 47 status detects locally running solr in 14393ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2167ms
ok 50 copying files around in 6532ms
ok 51 upconfig in 2647ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m23s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4869 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4869/

All tests passed

Build Log:
[...truncated 1253 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22328ms
ok 43 post solr log into solr in 8965ms
ok 44 start solr with ssl in 20208ms
ok 45 start solr with ssl and auth in 22539ms
ok 46 SOLR-11740 check 'solr stop' connection in 28418ms
ok 47 status detects locally running solr in 14372ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2174ms
ok 50 copying files around in 6511ms
ok 51 upconfig in 2723ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 4s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m38s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4868 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4868/

2 tests failed.
FAILED:  org.apache.solr.ltr.model.TestNeuralNetworkModel.layerDimensionMismatchTest

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([24587301CA150E9E]:0)


FAILED:  org.apache.solr.ltr.model.TestNeuralNetworkModel.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([24587301CA150E9E]:0)




Build Log:
[...truncated 1245 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22799ms
ok 43 post solr log into solr in 9100ms
ok 44 start solr with ssl in 20063ms
ok 45 start solr with ssl and auth in 22438ms
ok 46 SOLR-11740 check 'solr stop' connection in 28462ms
ok 47 status detects locally running solr in 14712ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6614ms
ok 51 upconfig in 2666ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.model.TestNeuralNetworkModel > layerDimensionMismatchTest FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([24587301CA150E9E]:0)

org.apache.solr.ltr.model.TestNeuralNetworkModel > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([24587301CA150E9E]:0)

org.apache.solr.ltr.model.TestNeuralNetworkModel > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestNeuralNetworkModel.txt, copied below:
  1> 01:05:28.663 [TEST-TestNeuralNetworkModel.layerDimensionMismatchTest-seed#[24587301CA150E9E]] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> juil. 12, 2023 11:05:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> AVERTISSEMENT: Suite execution timed out: org.apache.solr.ltr.model.TestNeuralNetworkModel
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@1aaacb19
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@1aaacb19
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@8c1022
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@8c1022
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@39093944
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@39093944
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:58964 to /127.0.0.1:43669 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5070a543
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5070a543
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:58964 to /127.0.0.1:43669 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@751c1214
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@751c1214
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:58964 to /127.0.0.1:43669 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@15f11195
  2> 	- locked sun.nio.ch.EPollSelectorImpl@65c2d976
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestNeuralNetworkModel-seed#[24587301CA150E9E]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@304d3230
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestNeuralNetworkModel.layerDimensionMismatchTest-seed#[24587301CA150E9E]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@9345e3d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@9345e3d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@25541a9f
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@3834da19
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.model.TestNeuralNetworkModel.setup(TestNeuralNetworkModel.java:57)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4b4ebd00
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@124bdaeb
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@174eaadf
  2> 
  2> "qtp1628590263-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@37ac54a6
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7bfc5c3a
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1628590263-18-acceptor-0@6714f6ce-ServerConnector@2284bd27{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:45907}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@198619c8
  2> 
  2> "qtp1628590263-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1628590263-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1628590263-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1628590263-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1628590263-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1628590263-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@32686973
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-9c059ba-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67b402fc
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@67b402fc
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7e308fbc
  2> 	- locked sun.nio.ch.EPollSelectorImpl@575191eb
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@779ca0bb
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@26034983
  2> 	- locked sun.nio.ch.EPollSelectorImpl@61e70c5d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@53da9467
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1b5022b2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@405526fa
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@1ea51e7
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7c263022
  2> 	- locked sun.nio.ch.EPollSelectorImpl@25fa5e1f
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@624480a9
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@3fa5ada0
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@3fa5ada0
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@562e3e58
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@562e3e58
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bc4a39f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bc4a39f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([24587301CA150E9E]:0)
  2> NOTE: reproduce with: gradlew test --tests TestNeuralNetworkModel.layerDimensionMismatchTest -Dtests.seed=24587301CA150E9E -Dtests.locale=fr-BE -Dtests.timezone=America/Port_of_Spain -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> juil. 12, 2023 11:05:26 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> AVERTISSEMENT: Will linger awaiting termination of 16 leaked thread(s).
  2> juil. 12, 2023 11:05:27 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> GRAVE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.model.TestNeuralNetworkModel: 
  2>    1) Thread[id=22, name=qtp1628590263-22, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=25, name=Session-HouseKeeper-9c059ba-1, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=19, name=qtp1628590263-19, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>    9) Thread[id=24, name=qtp1628590263-24, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=15, name=TEST-TestNeuralNetworkModel.layerDimensionMismatchTest-seed#[24587301CA150E9E], state=WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.model.TestNeuralNetworkModel.setup(TestNeuralNetworkModel.java:57)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=17, name=qtp1628590263-17, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=18, name=qtp1628590263-18-acceptor-0@6714f6ce-ServerConnector@2284bd27{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:45907}, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=21, name=qtp1628590263-21, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=20, name=qtp1628590263-20, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=23, name=qtp1628590263-23, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> juil. 12, 2023 11:05:27 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFOS: Starting to interrupt leaked threads:
  2>    1) Thread[id=22, name=qtp1628590263-22, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>    2) Thread[id=25, name=Session-HouseKeeper-9c059ba-1, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>    3) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>    4) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>    5) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestNeuralNetworkModel]
  2>    6) Thread[id=19, name=qtp1628590263-19, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>    8) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestNeuralNetworkModel]
  2>    9) Thread[id=24, name=qtp1628590263-24, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>   10) Thread[id=15, name=TEST-TestNeuralNetworkModel.layerDimensionMismatchTest-seed#[24587301CA150E9E], state=WAITING, group=TGRP-TestNeuralNetworkModel]
  2>   11) Thread[id=17, name=qtp1628590263-17, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>   12) Thread[id=18, name=qtp1628590263-18-acceptor-0@6714f6ce-ServerConnector@2284bd27{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:45907}, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>   13) Thread[id=21, name=qtp1628590263-21, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>   14) Thread[id=20, name=qtp1628590263-20, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  2>   15) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestNeuralNetworkModel]
  2>   16) Thread[id=23, name=qtp1628590263-23, state=TIMED_WAITING, group=TGRP-TestNeuralNetworkModel]
  1> 03:05:27.915 [TEST-TestNeuralNetworkModel.layerDimensionMismatchTest-seed#[24587301CA150E9E]] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:571) ~[solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.model.TestNeuralNetworkModel.setup(TestNeuralNetworkModel.java:57) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/modules/ltr/build/tmp/tests-tmp/solr.ltr.model.TestNeuralNetworkModel_24587301CA150E9E-001
  2> Jul 13, 2023 3:05:28 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([24587301CA150E9E]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=34, maxMBSortInHeap=6.532675296075232, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=fr-BE, timezone=America/Port_of_Spain
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=253255872,total=324534272
  2> NOTE: All tests run in this JVM: [TestNeuralNetworkModel]
  2> NOTE: reproduce with: gradlew test --tests TestNeuralNetworkModel -Dtests.seed=24587301CA150E9E -Dtests.locale=fr-BE -Dtests.timezone=America/Port_of_Spain -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 180 test(s), 2 failure(s)

180 tests completed, 2 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.model.TestNeuralNetworkModel.layerDimensionMismatchTest (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestNeuralNetworkModel.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.model.TestNeuralNetworkModel.layerDimensionMismatchTest" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=24587301CA150E9E -Ptests.file.encoding=US-ASCII

  - org.apache.solr.ltr.model.TestNeuralNetworkModel.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestNeuralNetworkModel.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.model.TestNeuralNetworkModel.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=24587301CA150E9E -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 29s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m56.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4867 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4867/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22280ms
ok 43 post solr log into solr in 8850ms
ok 44 start solr with ssl in 19920ms
ok 45 start solr with ssl and auth in 22792ms
ok 46 SOLR-11740 check 'solr stop' connection in 28043ms
ok 47 status detects locally running solr in 14372ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2202ms
ok 50 copying files around in 6522ms
ok 51 upconfig in 2614ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 3s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m37s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4866 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4866/

1 tests failed.
FAILED:  org.apache.solr.TestTolerantSearch.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([8B4A67A10A8DABC9]:0)




Build Log:
[...truncated 1226 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22357ms
ok 43 post solr log into solr in 9302ms
ok 44 start solr with ssl in 20075ms
ok 45 start solr with ssl and auth in 22170ms
ok 46 SOLR-11740 check 'solr stop' connection in 28147ms
ok 47 status detects locally running solr in 14427ms
ok 48 status does not expose cli parameters to end user in 217ms
ok 49 listing out files in 2196ms
ok 50 copying files around in 6766ms
ok 51 upconfig in 2698ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.TestTolerantSearch > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([8B4A67A10A8DABC9]:0)

org.apache.solr.TestTolerantSearch > test suite's output saved to /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.TestTolerantSearch.txt, copied below:
  2> 3552 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 3593 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 3616 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/data-dir-1-001
  2> 3725 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 3770 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 3806 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 3853 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 ####initCore
  2> 3855 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 ####initCore end
  2> 3857 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.SolrTestCaseJ4 Writing core.properties file to /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-002/cores/core
  2> 4487 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 4490 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 4501 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 4623 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 4655 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1c4da730{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 4685 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.AbstractConnector Started ServerConnector@756872b1{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:٤٠٧٦٣}
  2> 4689 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.Server Started Server@3e816a8d{STARTING}[10.0.15,sto=٠] @٤٧٠٦ms
  2> 4709 ERROR (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 4710 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 4718 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.4.0
  2> 4729 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 4730 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 4730 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-12T18:42:35.614540Z
  2> 4753 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-001 (source: servlet config: solr.solr.home)
  2> 4775 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-001/solr.xml
  2> 6951 WARN  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 7337 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-002/cores
  2> 7339 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 7679 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 7699 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.7.0
  2> 7894 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=test
  2> 8472 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.D.s.WordDelimiterFilterFactory Solr loaded a deprecated plugin/analysis class [solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
  2> 8530 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.D.s.CurrencyField Solr loaded a deprecated plugin/analysis class [solr.CurrencyField]. Please consult documentation how to replace it accordingly.
  2> 9105 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
  2> 9108 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 9438 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-001/collection1, trusted=true
  2> 9563 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, configSetBaseDir=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-001, hostPort=40763, coreRootDirectory=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-002/cores}
  2> 9574 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-002/cores/core], dataDir=[/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-002/cores/core/data/]
  2> 9676 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=39, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=true, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.8128083780044583]
  2> 10633 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 10720 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 10721 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 10737 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=36, maxMergedSegmentMB=32.88987636566162, floorSegmentMB=0.6732807159423828, forceMergeDeletesPctAllowed=10.119658692326478, segmentsPerTier=38.0, maxCFSSegmentSizeMB=0.3517284393310547, noCFSRatio=0.4113642783627721, deletesPctAllowed=39.632537327725885
  2> 10790 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_8B4A67A10A8DABC9-001/tempDir-001/collection1/conf
  2> 10810 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 10827 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> يوليو ١٣, ٢٠٢٣ ٤:٤٢:٣٤ ص com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.TestTolerantSearch
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@61682861
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@61682861
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@3cc86edd
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@3cc86edd
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@967f191
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@967f191
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:56320 to /127.0.0.1:35757 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188d5ba
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@188d5ba
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@306c304e
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@1aa7ecca
  2> 
  2> "/127.0.0.1:56320 to /127.0.0.1:35757 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ec9fdd0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7ec9fdd0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@76505305
  2> 
  2> "/127.0.0.1:56320 to /127.0.0.1:35757 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@489460e1
  2> 	- locked sun.nio.ch.EPollSelectorImpl@192bd7fd
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@71d44a3
  2> 
  2> "SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@5ec9e467
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@7e68f9ba
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@7e68f9ba
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@6fdfb513
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@4274361
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100)
  2> 	at app//org.apache.solr.TestTolerantSearch.createThings(TestTolerantSearch.java:64)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4374e827
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@528fc6f2
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@406a869d
  2> 
  2> "qtp1802955843-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1a391380
  2> 	- locked sun.nio.ch.EPollSelectorImpl@362dfe63
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1802955843-18-acceptor-٠@7ac9a08e-ServerConnector@756872b1{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:٤٠٧٦٣}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2267d855
  2> 
  2> "qtp1802955843-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1802955843-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1802955843-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1802955843-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1802955843-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1802955843-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1bfb408a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-635860c6-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@498ce566
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@498ce566
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@108bd4d2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@cb213e6
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@2cb7ff90
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4d0cfdea
  2> 	- locked sun.nio.ch.EPollSelectorImpl@638586f8
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@26e56d98
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2d66e31e
  2> 	- locked sun.nio.ch.EPollSelectorImpl@31d7f877
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@6c52a4b3
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1b1eb00a
  2> 	- locked sun.nio.ch.EPollSelectorImpl@54e347d7
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@332cf395
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@4e072046
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@4e072046
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@409d6db1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@409d6db1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2eed1b48
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2eed1b48
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
  2> يوليو ١٣, ٢٠٢٣ ٤:٤٢:٣٤ ص com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> يوليو ١٣, ٢٠٢٣ ٤:٤٢:٣٥ ص com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.TestTolerantSearch: 
  2>    1) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=23, name=qtp1802955843-23, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=21, name=qtp1802955843-21, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=24, name=qtp1802955843-24, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x000000084040b040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=19, name=qtp1802955843-19, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=18, name=qtp1802955843-18-acceptor-٠@7ac9a08e-ServerConnector@756872b1{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:٤٠٧٦٣}, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=20, name=qtp1802955843-20, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=22, name=qtp1802955843-22, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=25, name=Session-HouseKeeper-635860c6-1, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=17, name=qtp1802955843-17, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033d040.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=15, name=SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker, state=WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100)
  2>         at app//org.apache.solr.TestTolerantSearch.createThings(TestTolerantSearch.java:64)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> يوليو ١٣, ٢٠٢٣ ٤:٤٢:٣٥ ص com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    2) Thread[id=23, name=qtp1802955843-23, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>    3) Thread[id=21, name=qtp1802955843-21, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    5) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    6) Thread[id=24, name=qtp1802955843-24, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    8) Thread[id=19, name=qtp1802955843-19, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>    9) Thread[id=18, name=qtp1802955843-18-acceptor-٠@7ac9a08e-ServerConnector@756872b1{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:٤٠٧٦٣}, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>   10) Thread[id=20, name=qtp1802955843-20, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   11) Thread[id=22, name=qtp1802955843-22, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   12) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestTolerantSearch]
  2>   13) Thread[id=25, name=Session-HouseKeeper-635860c6-1, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   14) Thread[id=17, name=qtp1802955843-17, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>   15) Thread[id=15, name=SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker, state=WAITING, group=TGRP-TestTolerantSearch]
  2>   16) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestTolerantSearch]
  2> 7204306 ERROR (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter.
  2>           => org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:571)
  2> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:571) ~[main/:?]
  2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [main/:?]
  2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 28cbd64787f9852f3a8dae15039383f235235632 [snapshot build, details omitted]]
  2> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.4.0-SNAPSHOT.jar:?]
  2> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100) [solr-test-framework-9.4.0-SNAPSHOT.jar:?]
  2> 	at org.apache.solr.TestTolerantSearch.createThings(TestTolerantSearch.java:64) [test/:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  2> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  2> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7204806 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.Server Stopped Server@3e816a8d{STOPPING}[10.0.15,sto=٠]
  2> 7204810 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.AbstractConnector Stopped ServerConnector@756872b1{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:٠}
  2> 7204813 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1c4da730{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,STOPPED}
  2> 7204819 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1187198471
  2> 7204821 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@5f101d74 collection1
  2> 7204822 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@5f101d74
  2> 7204847 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 7204852 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7204861 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7204865 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7204871 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 7204871 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> 7204872 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log messages
  2> 7204872 INFO  (SUITE-TestTolerantSearch-seed#[8B4A67A10A8DABC9]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching regex: ignore_exception
  2> Jul 12, 2023 8:42:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([8B4A67A10A8DABC9]:0)
  2> NOTE: test params are: codec=CheapBastard, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ar-001, timezone=Etc/GMT-8
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=214176352,total=324534272
  2> NOTE: All tests run in this JVM: [TestTolerantSearch]
  2> NOTE: reproduce with: gradlew test --tests TestTolerantSearch -Dtests.seed=8B4A67A10A8DABC9 -Dtests.locale=ar-001 -Dtests.timezone=Etc/GMT-8 -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4846 test(s), 1 failure(s), 203 skipped

4846 tests completed, 1 failed, 203 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.TestTolerantSearch.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.TestTolerantSearch.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.TestTolerantSearch.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=8B4A67A10A8DABC9 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 4m 36s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h5m12s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4865 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4865/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22621ms
ok 43 post solr log into solr in 9134ms
ok 44 start solr with ssl in 19851ms
ok 45 start solr with ssl and auth in 22365ms
ok 46 SOLR-11740 check 'solr stop' connection in 28603ms
ok 47 status detects locally running solr in 14382ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2164ms
ok 50 copying files around in 6527ms
ok 51 upconfig in 2606ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m19.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4864 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4864/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22357ms
ok 43 post solr log into solr in 9199ms
ok 44 start solr with ssl in 20497ms
ok 45 start solr with ssl and auth in 22519ms
ok 46 SOLR-11740 check 'solr stop' connection in 28162ms
ok 47 status detects locally running solr in 14408ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2174ms
ok 50 copying files around in 6564ms
ok 51 upconfig in 2651ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m24s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4863 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4863/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 21994ms
ok 43 post solr log into solr in 9028ms
ok 44 start solr with ssl in 19676ms
ok 45 start solr with ssl and auth in 22091ms
ok 46 SOLR-11740 check 'solr stop' connection in 28052ms
ok 47 status detects locally running solr in 14657ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2145ms
ok 50 copying files around in 6464ms
ok 51 upconfig in 2594ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m24s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4862 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4862/

2 tests failed.
FAILED:  org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([121F6D86B44C0DDC]:0)


FAILED:  org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([121F6D86B44C0DDC]:0)




Build Log:
[...truncated 1245 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22592ms
ok 43 post solr log into solr in 8914ms
ok 44 start solr with ssl in 21747ms
ok 45 start solr with ssl and auth in 22515ms
ok 46 SOLR-11740 check 'solr stop' connection in 28372ms
ok 47 status detects locally running solr in 14670ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2138ms
ok 50 copying files around in 6459ms
ok 51 upconfig in 2607ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.feature.TestEdisMaxSolrFeature > testEdisMaxSolrFeatureCustomMM FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([121F6D86B44C0DDC]:0)

org.apache.solr.ltr.feature.TestEdisMaxSolrFeature > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([121F6D86B44C0DDC]:0)

org.apache.solr.ltr.feature.TestEdisMaxSolrFeature > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.txt, copied below:
  1> 08:31:12.181 [TEST-TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM-seed#[121F6D86B44C0DDC]] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> Jul 12, 2023 10:31:10 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.ltr.feature.TestEdisMaxSolrFeature
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@7a901280
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@7a901280
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@1c013414
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@1c013414
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@1173effa
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@1173effa
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:58450 to /127.0.0.1:33401 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6026e650
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6026e650
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:58450 to /127.0.0.1:33401 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30c29cf7
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@30c29cf7
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:58450 to /127.0.0.1:33401 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@25f6bf6f
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2b748814
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestEdisMaxSolrFeature-seed#[121F6D86B44C0DDC]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@7c37ef4c
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM-seed#[121F6D86B44C0DDC]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@7af053ca
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@7af053ca
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@58b6f119
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@482b30a7
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.before(TestEdisMaxSolrFeature.java:30)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4710359e
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@41eefbf
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@6bcff81c
  2> 
  2> "qtp1875529487-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3c0a35cf
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4f1a56e6
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1875529487-18-acceptor-0@4b02272-ServerConnector@77c81a95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:38909}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5b015801
  2> 
  2> "qtp1875529487-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1875529487-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1875529487-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1875529487-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1875529487-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1875529487-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23172ea8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-9bc6539-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4e5ed348
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4e5ed348
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@53ee1103
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2fe8a2f
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@112e91ad
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@188696b6
  2> 	- locked sun.nio.ch.EPollSelectorImpl@20fad27d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@5c07032c
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3bfb968
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7566e4c2
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@41be5b78
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@59f9fffe
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2f66b786
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@4c475c42
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@51ec3ffb
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@51ec3ffb
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@13ee91f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@13ee91f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fe8a7c
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5fe8a7c
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([121F6D86B44C0DDC]:0)
  2> NOTE: reproduce with: gradlew test --tests TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM -Dtests.seed=121F6D86B44C0DDC -Dtests.locale=qu -Dtests.timezone=Etc/Zulu -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> Jul 12, 2023 10:31:10 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> Jul 12, 2023 10:31:12 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.feature.TestEdisMaxSolrFeature: 
  2>    1) Thread[id=15, name=TEST-TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM-seed#[121F6D86B44C0DDC], state=WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.before(TestEdisMaxSolrFeature.java:30)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=17, name=qtp1875529487-17, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=19, name=qtp1875529487-19, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=22, name=qtp1875529487-22, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=20, name=qtp1875529487-20, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x0000000840305c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$352/0x000000084040e440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=18, name=qtp1875529487-18-acceptor-0@4b02272-ServerConnector@77c81a95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:38909}, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=25, name=Session-HouseKeeper-9bc6539-1, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=21, name=qtp1875529487-21, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   15) Thread[id=24, name=qtp1875529487-24, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=23, name=qtp1875529487-23, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> Jul 12, 2023 10:31:12 A.M. com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=15, name=TEST-TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM-seed#[121F6D86B44C0DDC], state=WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>    2) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>    3) Thread[id=17, name=qtp1875529487-17, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>    4) Thread[id=19, name=qtp1875529487-19, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>    5) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>    6) Thread[id=22, name=qtp1875529487-22, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>    7) Thread[id=20, name=qtp1875529487-20, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>    8) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>    9) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>   10) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>   11) Thread[id=18, name=qtp1875529487-18-acceptor-0@4b02272-ServerConnector@77c81a95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:38909}, state=RUNNABLE, group=TGRP-TestEdisMaxSolrFeature]
  2>   12) Thread[id=25, name=Session-HouseKeeper-9bc6539-1, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>   13) Thread[id=21, name=qtp1875529487-21, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>   14) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>   15) Thread[id=24, name=qtp1875529487-24, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  2>   16) Thread[id=23, name=qtp1875529487-23, state=TIMED_WAITING, group=TGRP-TestEdisMaxSolrFeature]
  1> 10:31:12.014 [TEST-TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM-seed#[121F6D86B44C0DDC]] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:571) ~[solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.before(TestEdisMaxSolrFeature.java:30) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/modules/ltr/build/tmp/tests-tmp/solr.ltr.feature.TestEdisMaxSolrFeature_121F6D86B44C0DDC-001
  2> Jul 12, 2023 10:31:12 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([121F6D86B44C0DDC]:0)
  2> NOTE: test params are: codec=FastDecompressionCompressingStoredFieldsData(storedFieldsFormat=Lucene90CompressingStoredFieldsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=20425, maxDocsPerChunk=6, blockShift=9), termVectorsFormat=Lucene90CompressingTermVectorsFormat(compressionMode=FAST_DECOMPRESSION, chunkSize=20425, maxDocsPerChunk=6, blockSize=9)), sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=qu, timezone=Etc/Zulu
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=251521936,total=324534272
  2> NOTE: All tests run in this JVM: [TestEdisMaxSolrFeature]
  2> NOTE: reproduce with: gradlew test --tests TestEdisMaxSolrFeature -Dtests.seed=121F6D86B44C0DDC -Dtests.locale=qu -Dtests.timezone=Etc/Zulu -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 186 test(s), 2 failure(s)

186 tests completed, 2 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.testEdisMaxSolrFeatureCustomMM" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=121F6D86B44C0DDC -Ptests.file.encoding=US-ASCII

  - org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.feature.TestEdisMaxSolrFeature.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=121F6D86B44C0DDC -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 34s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h4m54s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4861 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4861/

1 tests failed.
FAILED:  org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([BC92566ABBF2A656]:0)




Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22955ms
ok 43 post solr log into solr in 9149ms
ok 44 start solr with ssl in 19844ms
ok 45 start solr with ssl and auth in 22343ms
ok 46 SOLR-11740 check 'solr stop' connection in 28109ms
ok 47 status detects locally running solr in 14531ms
ok 48 status does not expose cli parameters to end user in 220ms
ok 49 listing out files in 2212ms
ok 50 copying files around in 6688ms
ok 51 upconfig in 2698ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:solrj:test

org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([BC92566ABBF2A656]:0)

org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test > test suite's output saved to /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.txt, copied below:
  2> 3603 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: uniqueKey
  2> 4155 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 4161 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 4167 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> 4190 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/data-dir-1-001
  2> 4395 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
  2> 4453 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl="None")
  2> 4500 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 4583 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 ####initCore
  2> 4584 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 ####initCore end
  2> 4585 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.SolrTestCaseJ4 Writing core.properties file to /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-002/cores/core
  2> 5093 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 5096 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 5105 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 5236 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 5266 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@75fa6e86{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 5294 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.e.j.s.AbstractConnector Started ServerConnector@3fbc6e1a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33027}
  2> 5297 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.e.j.s.Server Started Server@7afdcad9{STARTING}[10.0.15,sto=0] @5331ms
  2> 5312 ERROR (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 5313 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 5321 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.4.0-SNAPSHOT 61e5a5b78b89fe0241e1c72184d764993b4e8c57 [snapshot build, details omitted]
  2> 5322 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 5322 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 5323 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-12T03:07:43.981131Z
  2> 5339 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001 (source: servlet config: solr.solr.home)
  2> 5359 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/solr.xml
  2> 7960 WARN  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 8813 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001
  2> 8817 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 9006 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Couldn't add files from /tmp/src/solr/solr/modules/extraction/lib filtered by .*\.jar to classpath: java.nio.file.NoSuchFileException: /tmp/src/solr/solr/modules/extraction/lib
  2> 9008 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Couldn't add files from /tmp/src/solr/solr/modules/clustering/lib filtered by .*\.jar to classpath: java.nio.file.NoSuchFileException: /tmp/src/solr/solr/modules/clustering/lib
  2> 9009 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Couldn't add files from /tmp/src/solr/solr/modules/langid/lib filtered by .*\.jar to classpath: java.nio.file.NoSuchFileException: /tmp/src/solr/solr/modules/langid/lib
  2> 9010 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Couldn't add files from /tmp/src/solr/solr/modules/ltr/lib filtered by .*\.jar to classpath: java.nio.file.NoSuchFileException: /tmp/src/solr/solr/modules/ltr/lib
  2> 9011 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Couldn't add files from /tmp/src/solr/solr/modules/scripting/lib filtered by .*\.jar to classpath: java.nio.file.NoSuchFileException: /tmp/src/solr/solr/modules/scripting/lib
  2> 9013 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 9013 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.7.0
  2> 9044 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.SolrIndexConfig IndexWriter infoStream solr logging is enabled
  2> 9224 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=example
  2> 10155 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 10328 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedWordSetResource for path /schema/analysis/stopwords/english
  2> 10330 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.RestManager Registered ManagedResource impl org.apache.solr.rest.schema.analysis.ManagedSynonymGraphFilterFactory$SynonymManager for path /schema/analysis/synonyms/english
  2> 10331 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.RestManager Added observer of type org.apache.solr.rest.schema.analysis.ManagedStopFilterFactory to existing ManagedResource /schema/analysis/stopwords/english
  2> 10331 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.RestManager Added observer of type org.apache.solr.rest.schema.analysis.ManagedSynonymGraphFilterFactory to existing ManagedResource /schema/analysis/synonyms/english
  2> 10350 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema example/1.6 with uniqueid field id
  2> 10446 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, configSetBaseDir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001, hostPort=33027, coreRootDirectory=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-002/cores}
  2> 10456 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1, trusted=true
  2> 10651 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1], dataDir=[/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1/data/]
  2> 10775 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-11-thread-1]: initDynamicDefaults maxThreadCount=2 maxMergeCount=7
  2> 10816 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: init: current segments file is "segments"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@3b3a3fe
  2> 10825 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 10850 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
  2> 10851 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 10859 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: 23 ms to checkpoint
  2> 10861 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: init: create=true reader=null
  2> 10883 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: 
  2> dir=MockDirectoryWrapper(ByteBuffersDirectory@4f80362 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@baba98f)
  2> index=
  2> version=9.7.0
  2> analyzer=org.apache.solr.update.SolrIndexConfig$DelayedSchemaAnalyzer
  2> ramBufferSizeMB=100.0
  2> maxBufferedDocs=-1
  2> mergedSegmentWarmer=null
  2> delPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper
  2> commit=null
  2> openMode=CREATE
  2> similarity=org.apache.solr.search.similarities.SchemaSimilarityFactory$SchemaSimilarity
  2> mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=2, maxMergeCount=7, ioThrottle=true
  2> codec=Lucene95
  2> infoStream=org.apache.solr.update.LoggingInfoStream
  2> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.1, deletesPctAllowed=20.0
  2> readerPooling=true
  2> perThreadHardLimitMB=1945
  2> useCompoundFile=false
  2> commitOnClose=true
  2> indexSort=null
  2> checkPendingFlushOnUpdate=true
  2> softDeletesField=null
  2> maxFullFlushMergeWaitMillis=500
  2> leafSorter=null
  2> eventListener=org.apache.lucene.index.IndexWriterEventListener$1@284b656d
  2> writer=org.apache.solr.update.SolrIndexWriter@332d6f18
  2> 
  2> 10886 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: MMapDirectory.UNMAP_SUPPORTED=true
  2> 10887 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: now flush at close
  2> 10887 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]:   start flush: applyAllDeletes=true
  2> 10888 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]:   index before flush 
  2> 10889 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: startFullFlush
  2> 10893 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: coreLoadExecutor-11-thread-1 finishFullFlush success=true
  2> 10902 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
  2> 10903 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreLoadExecutor-11-thread-1]: waitApply: no deletes to apply
  2> 10907 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-11-thread-1]: updateMergeThreads ioThrottle=true targetMBPerSec=10240.0 MB/sec
  2> 10908 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-11-thread-1]: now merge
  2> 10909 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-11-thread-1]:   index(source): 
  2> 10909 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-11-thread-1]:   no more merges pending; now return
  2> 10910 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: waitForMerges
  2> 10910 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: waitForMerges done
  2> 10911 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: start
  2> 10911 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: enter lock
  2> 10912 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: now prepare
  2> 10912 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: prepareCommit: flush
  2> 10913 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]:   index before flush 
  2> 10914 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: startFullFlush
  2> 10914 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
  2> 10914 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreLoadExecutor-11-thread-1]: waitApply: no deletes to apply
  2> 10915 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: coreLoadExecutor-11-thread-1 finishFullFlush success=true
  2> 10915 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: startCommit(): start
  2> 10916 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: startCommit index= changeCount=2
  2> 10928 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: startCommit: wrote pending segments file "pending_segments_1"
  2> 10930 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: done all syncs: []
  2> 10931 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: pendingCommit != null
  2> 10940 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: done writing segments file "segments_1"
  2> 10941 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now checkpoint "" [0 segments ; isCommit = true]
  2> 10952 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: 10 ms to checkpoint
  2> 10955 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 10956 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: took 43.2 msec
  2> 10956 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: commit: done
  2> 10958 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: rollback
  2> 10961 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: all running merges have aborted
  2> 10963 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: rollback: done finish merges
  2> 10964 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: abort
  2> 10966 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: done abort success=true
  2> 10967 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: rollback: infos=
  2> 10968 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
  2> 10968 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 10969 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: 1 ms to checkpoint
  2> 10970 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 11908 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 11992 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
  2> 11993 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
  2> 12018 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 
  2> 12029 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 12038 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreLoadExecutor-11-thread-1]: initDynamicDefaults maxThreadCount=2 maxMergeCount=7
  2> 12044 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: init: current segments file is "segments_1"; deletionPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper@3b3a3fe
  2> 12050 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: init: load commit "segments_1"
  2> 12051 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 12052 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
  2> 12053 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: now delete 0 files: []
  2> 12053 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreLoadExecutor-11-thread-1]: 1 ms to checkpoint
  2> 12053 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: init: create=false reader=null
  2> 12055 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: 
  2> dir=MockDirectoryWrapper(ByteBuffersDirectory@4f80362 lockFactory=org.apache.lucene.store.SingleInstanceLockFactory@baba98f)
  2> index=
  2> version=9.7.0
  2> analyzer=org.apache.solr.update.SolrIndexConfig$DelayedSchemaAnalyzer
  2> ramBufferSizeMB=100.0
  2> maxBufferedDocs=-1
  2> mergedSegmentWarmer=null
  2> delPolicy=org.apache.solr.core.IndexDeletionPolicyWrapper
  2> commit=null
  2> openMode=APPEND
  2> similarity=org.apache.solr.search.similarities.SchemaSimilarityFactory$SchemaSimilarity
  2> mergeScheduler=ConcurrentMergeScheduler: maxThreadCount=2, maxMergeCount=7, ioThrottle=true
  2> codec=Lucene95
  2> infoStream=org.apache.solr.update.LoggingInfoStream
  2> mergePolicy=[TieredMergePolicy: maxMergeAtOnce=10, maxMergedSegmentMB=5120.0, floorSegmentMB=2.0, forceMergeDeletesPctAllowed=10.0, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.1, deletesPctAllowed=20.0
  2> readerPooling=true
  2> perThreadHardLimitMB=1945
  2> useCompoundFile=false
  2> commitOnClose=true
  2> indexSort=null
  2> checkPendingFlushOnUpdate=true
  2> softDeletesField=null
  2> maxFullFlushMergeWaitMillis=500
  2> leafSorter=null
  2> eventListener=org.apache.lucene.index.IndexWriterEventListener$1@284b656d
  2> writer=org.apache.solr.update.SolrIndexWriter@69badb12
  2> 
  2> 12063 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: MMapDirectory.UNMAP_SUPPORTED=true
  2> 12064 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: flush at getReader
  2> 12068 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: startFullFlush
  2> 12069 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
  2> 12070 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreLoadExecutor-11-thread-1]: waitApply: no deletes to apply
  2> 12082 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: incRefDeleter for NRT reader version=2 segments=
  2> 12084 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: return reader version=2 reader=StandardDirectoryReader(segments_1:2:nrt)
  2> 12096 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MP][coreLoadExecutor-11-thread-1]: findMerges: 0 segments
  2> 12097 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreLoadExecutor-11-thread-1]: coreLoadExecutor-11-thread-1 finishFullFlush success=true
  2> 12100 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreLoadExecutor-11-thread-1]: getReader took 36 ms
  2> 12174 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1/conf
  2> 12179 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path _rest_managed.json using file:dir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1/conf
  2> 12180 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResource Loaded initArgs {} for /rest/managed
  2> 12182 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path _schema_analysis_stopwords_english.json using file:dir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1/conf
  2> 12183 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResource Loaded initArgs {ignoreCase=true} for /schema/analysis/stopwords/english
  2> 12183 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.s.a.ManagedWordSetResource Loaded 35 words for /schema/analysis/stopwords/english
  2> 12184 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResource Notified 2 observers of /schema/analysis/stopwords/english
  2> 12185 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage Loaded LinkedHashMap at path _schema_analysis_synonyms_english.json using file:dir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.embedded.SolrExampleStreamingHttp2Test_BC92566ABBF2A656-001/tempDir-001/collection1/conf
  2> 12186 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResource Loaded initArgs {ignoreCase=true, format=solr} for /schema/analysis/synonyms/english
  2> 12187 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.s.a.ManagedSynonymGraphFilterFactory Loaded 3 synonym mappings for /schema/analysis/synonyms/english
  2> 12192 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResource Notified 2 observers of /schema/analysis/synonyms/english
  2> 12220 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.DirectSolrSpellChecker init: {name=default, field=text, classname=solr.DirectSolrSpellChecker, distanceMeasure=internal, maxEdits=2, minPrefix=1, maxInspections=5, minQueryLength=4, accuracy=0.5, maxQueryFrequency=0.01}
  2> 12233 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.c.SuggestComponent Initializing SuggestComponent
  2> 12237 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.s.SolrSuggester init: {name=mySuggester, lookupImpl=FuzzyLookupFactory, dictionaryImpl=DocumentDictionaryFactory, field=cat, weightField=price, suggestAnalyzerFieldType=string, buildOnStartup=false}
  2> 12265 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.s.SolrSuggester Dictionary loaded with params: {name=mySuggester, lookupImpl=FuzzyLookupFactory, dictionaryImpl=DocumentDictionaryFactory, field=cat, weightField=price, suggestAnalyzerFieldType=string, buildOnStartup=false}
  2> 12290 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 12301 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1771182511575007232
  2> 12457 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.S.Request webapp=null path=null params={q=static+firstSearcher+warming+in+solrconfig.xml&distrib=false&event=firstSearcher} hits=0 status=0 QTime=145
  2> 12458 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.QuerySenderListener QuerySenderListener done.
  2> 12459 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: default
  2> 12459 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.h.c.SpellCheckComponent Loading spell index for spellchecker: wordbreak
  2> 12485 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> Mfu 12, 2023 7:07:42 LUMA LWA K com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@33e5af3b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@33e5af3b
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@2b85e3ff
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@2b85e3ff
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@4cbdba7b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@4cbdba7b
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:33988 to /127.0.0.1:44583 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1ce97e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1ce97e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@244e248f
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:189)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@62150f9e
  2> 
  2> "/127.0.0.1:33988 to /127.0.0.1:44583 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b827779
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7b827779
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "/127.0.0.1:33988 to /127.0.0.1:44583 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5a4d6410
  2> 	- locked sun.nio.ch.EPollSelectorImpl@f72598a
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@409bf450
  2> 
  2> "SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@173d1603
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@79d802aa
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@79d802aa
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@7fec9222
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@4eb0042b
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100)
  2> 	at app//org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.beforeTest(SolrExampleStreamingHttp2Test.java:43)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$196/0x0000000840257c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5393aa7e
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2ce12301
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@3707c93d
  2> 
  2> "qtp559651293-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@37d697b5
  2> 	- locked sun.nio.ch.EPollSelectorImpl@128a5365
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp559651293-18-acceptor-0@285b72df-ServerConnector@3fbc6e1a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33027}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1bcd3e28
  2> 
  2> "qtp559651293-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp559651293-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp559651293-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp559651293-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp559651293-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp559651293-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@22b9a2a9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-4751bf9d-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@21d1e274
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@21d1e274
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@61809c50
  2> 	- locked sun.nio.ch.EPollSelectorImpl@13cec236
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@207f66ed
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7c0d960c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@52e109bb
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@4acf403e
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@37666dab
  2> 	- locked sun.nio.ch.EPollSelectorImpl@525cc603
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@788f7401
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@427197b2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1807e296
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@c1d3941
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@1c08cf08
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@1c08cf08
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@28506e54
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@28506e54
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6652b0eb
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6652b0eb
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
  2> Mfu 12, 2023 7:07:42 LUMA LWA K com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> Mfu 12, 2023 7:07:43 LUMA LWA K com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test: 
  2>    1) Thread[id=25, name=Session-HouseKeeper-4751bf9d-1, state=TIMED_WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=22, name=qtp559651293-22, state=TIMED_WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x0000000840414440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=18, name=qtp559651293-18-acceptor-0@285b72df-ServerConnector@3fbc6e1a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33027}, state=RUNNABLE, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=19, name=qtp559651293-19, state=TIMED_WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=20, name=qtp559651293-20, state=TIMED_WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$358/0x0000000840414440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=21, name=qtp559651293-21, state=TIMED_WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=23, name=qtp559651293-23, state=TIMED_WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=15, name=SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker, state=WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100)
  2>         at app//org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.beforeTest(SolrExampleStreamingHttp2Test.java:43)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$196/0x0000000840257c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=17, name=qtp559651293-17, state=RUNNABLE, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033dc40.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-SolrExampleStreamingHttp2Test]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.g

[...truncated too long message...]

h success=true
  2> 7205429 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
  2> 7205429 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-19-thread-1]: waitApply: no deletes to apply
  2> 7205429 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-19-thread-1]: updateMergeThreads ioThrottle=true targetMBPerSec=10240.0 MB/sec
  2> 7205429 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-19-thread-1]: now merge
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-19-thread-1]:   index(source): 
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [MS][coreCloseExecutor-19-thread-1]:   no more merges pending; now return
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: waitForMerges
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: waitForMerges done
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: commit: start
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: commit: enter lock
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: commit: now prepare
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: prepareCommit: flush
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]:   index before flush 
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-19-thread-1]: startFullFlush
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: now apply all deletes for all segments buffered updates bytesUsed=0 reader pool bytesUsed=0
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [BD][coreCloseExecutor-19-thread-1]: waitApply: no deletes to apply
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-19-thread-1]: coreCloseExecutor-19-thread-1 finishFullFlush success=true
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: startCommit(): start
  2> 7205430 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]:   skip startCommit(): no changes pending
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-19-thread-1]: now delete 0 files: []
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: commit: pendingCommit == null; skip
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: commit: took 0.8 msec
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: commit: done
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: rollback
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: all running merges have aborted
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: rollback: done finish merges
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-19-thread-1]: abort
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [DW][coreCloseExecutor-19-thread-1]: done abort success=true
  2> 7205431 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IW][coreCloseExecutor-19-thread-1]: rollback: infos=
  2> 7205432 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-19-thread-1]: now checkpoint "" [0 segments ; isCommit = false]
  2> 7205432 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-19-thread-1]: now delete 0 files: []
  2> 7205432 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-19-thread-1]: 0 ms to checkpoint
  2> 7205432 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.LoggingInfoStream [IFD][coreCloseExecutor-19-thread-1]: now delete 0 files: []
  2> 7205440 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7205449 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7205453 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7205460 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log messages
  2> 7205460 INFO  (SUITE-SolrExampleStreamingHttp2Test-seed#[BC92566ABBF2A656]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching regex: ignore_exception
  2> Jul 12, 2023 5:07:44 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([BC92566ABBF2A656]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=864, maxMBSortInHeap=7.987579959584959, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=dav, timezone=Europe/Brussels
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=193259256,total=324534272
  2> NOTE: All tests run in this JVM: [SolrExampleStreamingHttp2Test]
  2> NOTE: reproduce with: gradlew test --tests SolrExampleStreamingHttp2Test -Dtests.seed=BC92566ABBF2A656 -Dtests.locale=dav -Dtests.timezone=Europe/Brussels -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:solrj:test (FAILURE): 1093 test(s), 1 failure(s), 39 skipped

1093 tests completed, 1 failed, 39 skipped

> Task :solr:solrj:test FAILED
> Task :solr:solrj:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.classMethod (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.embedded.SolrExampleStreamingHttp2Test.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=BC92566ABBF2A656 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:solrj:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/solrj/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 17s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m46s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4860 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4860/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22505ms
ok 43 post solr log into solr in 9743ms
ok 44 start solr with ssl in 20378ms
ok 45 start solr with ssl and auth in 22021ms
ok 46 SOLR-11740 check 'solr stop' connection in 28482ms
ok 47 status detects locally running solr in 14390ms
ok 48 status does not expose cli parameters to end user in 216ms
ok 49 listing out files in 2180ms
ok 50 copying files around in 6606ms
ok 51 upconfig in 2663ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 6s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m34.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4859 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4859/

2 tests failed.
FAILED:  org.apache.solr.cloud.SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([430326869D504C66]:0)


FAILED:  org.apache.solr.cloud.SolrCloudExampleTest.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([430326869D504C66]:0)




Build Log:
[...truncated 1231 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22763ms
ok 43 post solr log into solr in 9492ms
ok 44 start solr with ssl in 20274ms
ok 45 start solr with ssl and auth in 22847ms
ok 46 SOLR-11740 check 'solr stop' connection in 28243ms
ok 47 status detects locally running solr in 14481ms
ok 48 status does not expose cli parameters to end user in 225ms
ok 49 listing out files in 2201ms
ok 50 copying files around in 6680ms
ok 51 upconfig in 2683ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.cloud.SolrCloudExampleTest > testLoadDocsIntoGettingStartedCollection FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([430326869D504C66]:0)

org.apache.solr.cloud.SolrCloudExampleTest > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([430326869D504C66]:0)

org.apache.solr.cloud.SolrCloudExampleTest > test suite's output saved to /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.SolrCloudExampleTest.txt, copied below:
  2> 4358 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 4360 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 4396 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 4423 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.SolrCloudExampleTest_430326869D504C66-001/data-dir-1-001
  2> 4532 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 4581 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 4977 INFO  (SUITE-SolrCloudExampleTest-seed#[430326869D504C66]-worker) [] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
  2> 4997 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 4998 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> 5007 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 5026 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 0.0.0.0/0.0.0.0:0
  2> 5026 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting server
  2> 5095 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
  2> 5148 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer start zk server on port: 33133
  2> 5521 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 5587 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 5659 INFO  (zkConnectionManagerCallback-4-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 5661 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 5661 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 5692 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 5708 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 5735 INFO  (zkConnectionManagerCallback-6-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 5736 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 5737 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 5778 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
  2> 5794 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
  2> 5804 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
  2> 5816 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
  2> 5825 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
  2> 5835 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
  2> 5842 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
  2> 5853 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
  2> 5868 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
  2> 5877 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
  2> 5891 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
  2> 5912 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 5939 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Waiting up to 45000ms for client to connect to ZooKeeper
  2> 5959 INFO  (zkConnectionManagerCallback-10-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 5961 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 5962 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 6076 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
  2> 7075 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 7078 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 7097 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 7206 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 7261 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@699944ac{/,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 7313 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.s.AbstractConnector Started ServerConnector@7fe48bea{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:45091}
  2> 7317 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.s.Server Started Server@42dc59a7{STARTING}[10.0.15,sto=0] @7373ms
  2> 7332 ERROR (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 7334 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 7343 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.4.0
  2> 7345 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 7346 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 7347 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-11T20:20:14.306146Z
  2> 7372 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.SolrCloudExampleTest_430326869D504C66-001/control-001 (source: servlet config: solr.solr.home)
  2> 7396 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 7435 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 7520 INFO  (zkConnectionManagerCallback-12-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 7533 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 7534 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 7657 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
  2> 7658 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.SolrCloudExampleTest_430326869D504C66-001/control-001/solr.xml
  2> 10373 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.u.s.S.config Trusting all certificates configured for Client@65ddb449[provider=null,keyStore=null,trustStore=null]
  2> 10378 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@65ddb449[provider=null,keyStore=null,trustStore=null]
  2> 10641 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.u.s.S.config Trusting all certificates configured for Client@588be327[provider=null,keyStore=null,trustStore=null]
  2> 10642 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@588be327[provider=null,keyStore=null,trustStore=null]
  2> 10681 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:33133/solr
  2> 10702 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10719 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 10763 INFO  (zkConnectionManagerCallback-22-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 10764 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 10765 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 10895 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
  2> 10938 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 10967 INFO  (zkConnectionManagerCallback-24-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 10968 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11156 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 11239 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
  2> 11287 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:45091_
  2> 11299 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.Overseer Overseer (id=72058117657329669-127.0.0.1:45091_-n_0000000000) starting
  2> 11516 INFO  (OverseerStateUpdate-72058117657329669-127.0.0.1:45091_-n_0000000000) [n:127.0.0.1:45091_] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:45091_
  2> 11517 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45091_
  2> 11538 INFO  (OverseerStateUpdate-72058117657329669-127.0.0.1:45091_-n_0000000000) [n:127.0.0.1:45091_] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
  2> 11643 WARN  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 12507 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.SolrCloudExampleTest_430326869D504C66-001/control-001/cores
  2> 13851 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 14086 INFO  (TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]) [n:127.0.0.1:45091_] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/, solr.data.dir=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.SolrCloudExampleTest_430326869D504C66-001/tempDir-001/control/data, hostPort=45091, coreRootDirectory=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.SolrCloudExampleTest_430326869D504C66-001/control-001/cores, replicaType=NRT}
  2> KMS 11, 2023 5:20:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.cloud.SolrCloudExampleTest
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@78b6e8bc
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@78b6e8bc
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@5cef3f04
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@5cef3f04
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@50db41c4
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@50db41c4
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:48672 to /127.0.0.1:42041 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76b05cbc
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@76b05cbc
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@1fe7a06b
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@1aa7ecca
  2> 
  2> "/127.0.0.1:48672 to /127.0.0.1:42041 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53025a71
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53025a71
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@76505305
  2> 
  2> "/127.0.0.1:48672 to /127.0.0.1:42041 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@59ddf4b4
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7b1bf31c
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@71d44a3
  2> 
  2> "SUITE-SolrCloudExampleTest-seed#[430326869D504C66]" ID=16 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@fba057f
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]" ID=17 WAITING on java.util.concurrent.CountDownLatch$Sync@44ada36e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@44ada36e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:525)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:564)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@4eaa072b
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@78e4321e
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:347)
  2> 	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1161)
  2> 	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1134)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840213c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5c43bd0d
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@30dca07
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1d000612
  2> 
  2> "ZkTestServer Run Thread" ID=18 WAITING on org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@5c7d523
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@5c7d523
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:915)
  2> 	at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
  2> 	at app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-0" ID=20 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@68f76a6e
  2> 	- locked sun.nio.ch.EPollSelectorImpl@755ad1f4
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-1" ID=21 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5cc5bae2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@67101294
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0" ID=22 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@34e7b002
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1791021b
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2> 
  2> "ConnnectionExpirer" ID=19 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:554)
  2> 
  2> "NIOWorkerThread-1" ID=23 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66f2b8a1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66f2b8a1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "SessionTracker" ID=24 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:163)
  2> 
  2> "SyncThread:0" ID=25 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1118259
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1118259
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  2> 
  2> "ProcessThread(sid:0 cport:33133):" ID=26 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ff45d9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8ff45d9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2> 
  2> "RequestThrottler" ID=27 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17e10f07
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@17e10f07
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2> 
  2> "NIOWorkerThread-2" ID=28 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66f2b8a1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66f2b8a1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-SendThread(127.0.0.1:33133)" ID=29 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@64258aea
  2> 	- locked sun.nio.ch.EPollSelectorImpl@69abdf7d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2> 	at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2> 	at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-EventThread" ID=30 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b088e1c
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1b088e1c
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "NIOWorkerThread-3" ID=31 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66f2b8a1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66f2b8a1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "zkConnectionManagerCallback-4-thread-1" ID=32 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48e234d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@48e234d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-SendThread(127.0.0.1:33133)" ID=33 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4c348bec
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7b53f520
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2> 	at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2> 	at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-EventThread" ID=34 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2474fc38
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2474fc38
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-6-thread-1" ID=35 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@519537b0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@519537b0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-40" ID=40 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3996ad85
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5d08b562
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-41-acceptor-0@1c2b049e-ServerConnector@7fe48bea{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:45091}" ID=41 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5e9e07c7
  2> 
  2> "qtp1686135367-42" ID=42 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-43" ID=43 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-44" ID=44 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-45" ID=45 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-46" ID=46 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1686135367-47" ID=47 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@26ce8313
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-30064e37-1" ID=48 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a38e8ca
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2a38e8ca
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-17-thread-1" ID=52 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3203fb4a
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1c8d62f2
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@47c75fa
  2> 
  2> "httpShardExecutor-17-thread-2" ID=53 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4e89ac6b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@239a73c8
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@128886eb
  2> 
  2> "Connection evictor" ID=54 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=55 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-19-thread-1" ID=56 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@e6e155a
  2> 	- locked sun.nio.ch.EPollSelectorImpl@79836c97
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@45e014a8
  2> 
  2> "h2sc-19-thread-2" ID=57 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7aea6d38
  2> 	- locked sun.nio.ch.EPollSelectorImpl@a9e559e
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@364c2036
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-SendThread(127.0.0.1:33133)" ID=61 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@74a45552
  2> 	- locked sun.nio.ch.EPollSelectorImpl@16a17885
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2> 	at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2> 	at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-EventThread" ID=62 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f0b54c4
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f0b54c4
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-24-thread-1" ID=63 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47c1c41e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47c1c41e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "OverseerStateUpdate-72058117657329669-127.0.0.1:45091_-n_0000000000" ID=64 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11191771
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@11191771
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2> 	at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "OverseerCollectionConfigSetProcessor-72058117657329669-127.0.0.1:45091_-n_0000000000" ID=65 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c612087
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c612087
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2> 	at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
  2> 	at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "CloserThread" ID=67 WAITING on java.lang.Object@69a9f8bb
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@69a9f8bb
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([430326869D504C66]:0)
  2> NOTE: reproduce with: gradlew test --tests SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection -Dtests.seed=430326869D504C66 -Dtests.locale=nyn -Dtests.timezone=America/Winnipeg -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> KMS 11, 2023 5:20:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 36 leaked thread(s).
  2> KMS 11, 2023 5:20:12 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 36 threads leaked from SUITE scope at org.apache.solr.cloud.SolrCloudExampleTest: 
  2>    1) Thread[id=65, name=OverseerCollectionConfigSetProcessor-72058117657329669-127.0.0.1:45091_-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2>         at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
  2>         at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=32, name=zkConnectionManagerCallback-4-thread-1, state=WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=42, name=qtp1686135367-42, state=TIMED_WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=21, name=NIOServerCxnFactory.SelectorThread-1, state=RUNNABLE, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2>         at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2>    5) Thread[id=34, name=TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-EventThread, state=WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2>    6) Thread[id=44, name=qtp1686135367-44, state=TIMED_WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=41, name=qtp1686135367-41-acceptor-0@1c2b049e-ServerConnector@7fe48bea{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:45091}, state=RUNNABLE, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=45, name=qtp1686135367-45, state=TIMED_WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=48, name=Session-HouseKeeper-30064e37-1, state=TIMED_WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=27, name=RequestThrottler, state=WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2>   11) Thread[id=28, name=NIOWorkerThread-2, state=WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=33, name=TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-SendThread(127.0.0.1:33133), state=RUNNABLE, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2>         at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2>         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2>   13) Thread[id=26, name=ProcessThread(sid:0 cport:33133):, state=WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2>   14) Thread[id=57, name=h2sc-19-thread-2, state=RUNNABLE, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>   

[...truncated too long message...]

ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=63, name=zkConnectionManagerCallback-24-thread-1, state=WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=61, name=TEST-SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection-seed#[430326869D504C66]-SendThread(127.0.0.1:33133), state=TIMED_WAITING, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>         at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:362)
  2>         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
  2>    5) Thread[id=53, name=httpShardExecutor-17-thread-2, state=RUNNABLE, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=57, name=h2sc-19-thread-2, state=RUNNABLE, group=TGRP-SolrCloudExampleTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$330/0x00000008403efc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$264/0x000000084035e840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([430326869D504C66]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=1704, maxMBSortInHeap=7.453244654213121, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=nyn, timezone=America/Winnipeg
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=10,free=269258416,total=337641472
  2> NOTE: All tests run in this JVM: [SolrCliUptimeTest, SolrCloudExampleTest]
  2> NOTE: reproduce with: gradlew test --tests SolrCloudExampleTest -Dtests.seed=430326869D504C66 -Dtests.locale=nyn -Dtests.timezone=America/Winnipeg -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4848 test(s), 2 failure(s), 251 skipped

4848 tests completed, 2 failed, 251 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.SolrCloudExampleTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.SolrCloudExampleTest.testLoadDocsIntoGettingStartedCollection" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=430326869D504C66 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.cloud.SolrCloudExampleTest.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.SolrCloudExampleTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.SolrCloudExampleTest.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=430326869D504C66 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 3m 57s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h4m22s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4858 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4858/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22722ms
ok 43 post solr log into solr in 9017ms
ok 44 start solr with ssl in 19905ms
ok 45 start solr with ssl and auth in 22308ms
ok 46 SOLR-11740 check 'solr stop' connection in 28452ms
ok 47 status detects locally running solr in 14400ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2160ms
ok 50 copying files around in 6528ms
ok 51 upconfig in 2620ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m27s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4857 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4857/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22248ms
ok 43 post solr log into solr in 9390ms
ok 44 start solr with ssl in 20397ms
ok 45 start solr with ssl and auth in 21906ms
ok 46 SOLR-11740 check 'solr stop' connection in 28031ms
ok 47 status detects locally running solr in 14358ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2152ms
ok 50 copying files around in 6625ms
ok 51 upconfig in 2682ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 17s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m53.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4856 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4856/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22335ms
ok 43 post solr log into solr in 8918ms
ok 44 start solr with ssl in 19670ms
ok 45 start solr with ssl and auth in 22402ms
ok 46 SOLR-11740 check 'solr stop' connection in 28319ms
ok 47 status detects locally running solr in 14399ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2159ms
ok 50 copying files around in 6488ms
ok 51 upconfig in 2597ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m52s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4855 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4855/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22096ms
ok 43 post solr log into solr in 8845ms
ok 44 start solr with ssl in 19892ms
ok 45 start solr with ssl and auth in 22619ms
ok 46 SOLR-11740 check 'solr stop' connection in 28300ms
ok 47 status detects locally running solr in 14348ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2156ms
ok 50 copying files around in 6473ms
ok 51 upconfig in 2596ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 48s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m17s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4854 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4854/

All tests passed

Build Log:
[...truncated 1244 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22211ms
ok 43 post solr log into solr in 9537ms
ok 44 start solr with ssl in 20601ms
ok 45 start solr with ssl and auth in 22490ms
ok 46 SOLR-11740 check 'solr stop' connection in 28358ms
ok 47 status detects locally running solr in 14402ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2177ms
ok 50 copying files around in 6516ms
ok 51 upconfig in 2606ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 12s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 19m40.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4853 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4853/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22098ms
ok 43 post solr log into solr in 8886ms
ok 44 start solr with ssl in 19888ms
ok 45 start solr with ssl and auth in 21792ms
ok 46 SOLR-11740 check 'solr stop' connection in 28532ms
ok 47 status detects locally running solr in 14615ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6456ms
ok 51 upconfig in 2583ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m22.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4852 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4852/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22492ms
ok 43 post solr log into solr in 9145ms
ok 44 start solr with ssl in 19952ms
ok 45 start solr with ssl and auth in 22750ms
ok 46 SOLR-11740 check 'solr stop' connection in 28285ms
ok 47 status detects locally running solr in 14464ms
ok 48 status does not expose cli parameters to end user in 222ms
ok 49 listing out files in 2221ms
ok 50 copying files around in 6663ms
ok 51 upconfig in 2663ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 13s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m44.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4851 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4851/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22888ms
ok 43 post solr log into solr in 9199ms
ok 44 start solr with ssl in 19899ms
ok 45 start solr with ssl and auth in 22514ms
ok 46 SOLR-11740 check 'solr stop' connection in 28259ms
ok 47 status detects locally running solr in 14530ms
ok 48 status does not expose cli parameters to end user in 228ms
ok 49 listing out files in 2225ms
ok 50 copying files around in 6755ms
ok 51 upconfig in 2701ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 8s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m40.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4850 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4850/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22737ms
ok 43 post solr log into solr in 9124ms
ok 44 start solr with ssl in 20119ms
ok 45 start solr with ssl and auth in 22502ms
ok 46 SOLR-11740 check 'solr stop' connection in 28250ms
ok 47 status detects locally running solr in 14429ms
ok 48 status does not expose cli parameters to end user in 216ms
ok 49 listing out files in 2189ms
ok 50 copying files around in 6612ms
ok 51 upconfig in 2655ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 18s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m42s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4849 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4849/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22411ms
ok 43 post solr log into solr in 9181ms
ok 44 start solr with ssl in 20218ms
ok 45 start solr with ssl and auth in 22915ms
ok 46 SOLR-11740 check 'solr stop' connection in 28184ms
ok 47 status detects locally running solr in 14457ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2211ms
ok 50 copying files around in 6635ms
ok 51 upconfig in 2661ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 5s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4848 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4848/

All tests passed

Build Log:
[...truncated 1249 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22852ms
ok 43 post solr log into solr in 9340ms
ok 44 start solr with ssl in 20962ms
ok 45 start solr with ssl and auth in 22596ms
ok 46 SOLR-11740 check 'solr stop' connection in 28209ms
ok 47 status detects locally running solr in 14425ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2177ms
ok 50 copying files around in 6595ms
ok 51 upconfig in 2618ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 18s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m39.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4847 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4847/

All tests passed

Build Log:
[...truncated 1246 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22685ms
ok 43 post solr log into solr in 9052ms
ok 44 start solr with ssl in 19791ms
ok 45 start solr with ssl and auth in 22497ms
ok 46 SOLR-11740 check 'solr stop' connection in 28590ms
ok 47 status detects locally running solr in 14429ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2176ms
ok 50 copying files around in 6595ms
ok 51 upconfig in 2688ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 9s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m35s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4846 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4846/

All tests passed

Build Log:
[...truncated 1243 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22377ms
ok 43 post solr log into solr in 9140ms
ok 44 start solr with ssl in 20258ms
ok 45 start solr with ssl and auth in 22541ms
ok 46 SOLR-11740 check 'solr stop' connection in 28046ms
ok 47 status detects locally running solr in 14692ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2158ms
ok 50 copying files around in 6583ms
ok 51 upconfig in 2631ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 2s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m36.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4845 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4845/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22374ms
ok 43 post solr log into solr in 8710ms
ok 44 start solr with ssl in 19862ms
ok 45 start solr with ssl and auth in 22366ms
ok 46 SOLR-11740 check 'solr stop' connection in 28047ms
ok 47 status detects locally running solr in 14458ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6524ms
ok 51 upconfig in 2620ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m46.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4844 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4844/

All tests passed

Build Log:
[...truncated 1253 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22493ms
ok 43 post solr log into solr in 9343ms
ok 44 start solr with ssl in 20292ms
ok 45 start solr with ssl and auth in 22053ms
ok 46 SOLR-11740 check 'solr stop' connection in 28130ms
ok 47 status detects locally running solr in 14422ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2170ms
ok 50 copying files around in 6537ms
ok 51 upconfig in 2609ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4843 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4843/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23120ms
ok 43 post solr log into solr in 9403ms
ok 44 start solr with ssl in 20865ms
ok 45 start solr with ssl and auth in 22280ms
ok 46 SOLR-11740 check 'solr stop' connection in 28259ms
ok 47 status detects locally running solr in 14448ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6590ms
ok 51 upconfig in 2654ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m28s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4842 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4842/

2 tests failed.
FAILED:  org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([A62F678E68998A14]:0)


FAILED:  org.apache.solr.client.solrj.TestLBHttp2SolrClient.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([A62F678E68998A14]:0)




Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22617ms
ok 43 post solr log into solr in 9178ms
ok 44 start solr with ssl in 19929ms
ok 45 start solr with ssl and auth in 22848ms
ok 46 SOLR-11740 check 'solr stop' connection in 28078ms
ok 47 status detects locally running solr in 14417ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6557ms
ok 51 upconfig in 2635ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:solrj:test

org.apache.solr.client.solrj.TestLBHttp2SolrClient > testTwoServers FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([A62F678E68998A14]:0)

org.apache.solr.client.solrj.TestLBHttp2SolrClient > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([A62F678E68998A14]:0)

org.apache.solr.client.solrj.TestLBHttp2SolrClient > test suite's output saved to /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttp2SolrClient.txt, copied below:
  2> 3245 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 3280 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 3311 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/data-dir-1-001
  2> 3413 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 3454 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 3484 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 3517 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.SolrTestCaseJ4 ###Starting testTwoServers
  2> 4367 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 4370 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 4399 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 4527 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 4573 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6f47bc64{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 4599 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.AbstractConnector Started ServerConnector@1fcdb038{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33009}
  2> 4605 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.Server Started Server@a77751{STARTING}[10.0.15,sto=0] @4636ms
  2> 4629 ERROR (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 4632 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 4643 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]
  2> 4644 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 4645 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 4646 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-09T23:29:35.009686Z
  2> 4648 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001 (source: servlet config: solr.solr.home)
  2> 4670 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001/solr.xml
  2> 8695 WARN  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 9085 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001
  2> 9087 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 9178 WARN  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 9185 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.7.0
  2> 9558 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=test
  2> 9603 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
  2> 9606 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001/collection1, trusted=true
  2> 9710 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001/collection1], dataDir=[/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001/collection1/data/]
  2> 10789 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 10849 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 11018 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 11019 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 11058 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, solrconfig=bad_solrconfig.xml, solr.data.dir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001/collection1/data, hostPort=33009}
  2> 11215 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001/instance-0-001/collection1/conf
  2> 11238 INFO  (coreLoadExecutor-12-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 11269 INFO  (searcherExecutor-14-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> juil. 09, 2023 6:29:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> AVERTISSEMENT: Suite execution timed out: org.apache.solr.client.solrj.TestLBHttp2SolrClient
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@34376d25
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@34376d25
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@1367d0d2
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@1367d0d2
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@204866ae
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@204866ae
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:55202 to /127.0.0.1:42775 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@103e5c80
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@103e5c80
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@575da62
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:189)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@62150f9e
  2> 
  2> "/127.0.0.1:55202 to /127.0.0.1:42775 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3df2ef15
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3df2ef15
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "/127.0.0.1:55202 to /127.0.0.1:42775 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5e6c9313
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7561f476
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@409bf450
  2> 
  2> "SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@17d96c06
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@6cacf2c2
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@6cacf2c2
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:508)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:547)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@3d79b3af
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@7b75b18
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.client.solrj.TestLBHttp2SolrClient$SolrInstance.startJetty(TestLBHttp2SolrClient.java:327)
  2> 	at app//org.apache.solr.client.solrj.TestLBHttp2SolrClient.setUp(TestLBHttp2SolrClient.java:91)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7d8fcddc
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@216a2737
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@51fe0c7e
  2> 
  2> "h2sc-2-thread-1" ID=16 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6f4e33a6
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1f6df2e7
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@40aa89d2
  2> 
  2> "h2sc-2-thread-2" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3e519088
  2> 	- locked sun.nio.ch.EPollSelectorImpl@35605f62
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@762a39b7
  2> 
  2> "qtp2107975691-19" ID=19 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2bad996b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5b27bc33
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp2107975691-20-acceptor-0@21dc258-ServerConnector@1fcdb038{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33009}" ID=20 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@69c0a20f
  2> 
  2> "qtp2107975691-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp2107975691-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp2107975691-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp2107975691-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp2107975691-25" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp2107975691-26" ID=26 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5eec6b1a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-588b38bb-1" ID=27 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@194dc76d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@194dc76d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-7-thread-1" ID=28 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7f6d3653
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1351336b
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@73fdb84b
  2> 
  2> "httpShardExecutor-7-thread-2" ID=29 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2ed82ea0
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5959e2b9
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@4d073c3f
  2> 
  2> "Connection evictor" ID=30 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=31 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-9-thread-1" ID=32 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4f90986f
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5540309d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@25e69298
  2> 
  2> "h2sc-9-thread-2" ID=33 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3a82b9bb
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2c58d878
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@635a95d
  2> 
  2> "CloserThread" ID=35 WAITING on java.lang.Object@67c07e7c
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@67c07e7c
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=38 WAITING on java.util.concurrent.ForkJoinPool@33506ec1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@33506ec1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-14-thread-1" ID=39 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c304671
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1c304671
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([A62F678E68998A14]:0)
  2> NOTE: reproduce with: gradlew test --tests TestLBHttp2SolrClient.testTwoServers -Dtests.seed=A62F678E68998A14 -Dtests.locale=fr-GF -Dtests.timezone=America/Hermosillo -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> juil. 09, 2023 6:29:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> AVERTISSEMENT: Will linger awaiting termination of 18 leaked thread(s).
  2> juil. 09, 2023 6:29:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> GRAVE: 18 threads leaked from SUITE scope at org.apache.solr.client.solrj.TestLBHttp2SolrClient: 
  2>    1) Thread[id=28, name=httpShardExecutor-7-thread-1, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=21, name=qtp2107975691-21, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=33, name=h2sc-9-thread-2, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=15, name=TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14], state=WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:508)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:547)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.client.solrj.TestLBHttp2SolrClient$SolrInstance.startJetty(TestLBHttp2SolrClient.java:327)
  2>         at app//org.apache.solr.client.solrj.TestLBHttp2SolrClient.setUp(TestLBHttp2SolrClient.java:91)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=22, name=qtp2107975691-22, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=25, name=qtp2107975691-25, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=29, name=httpShardExecutor-7-thread-2, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=35, name=CloserThread, state=WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>    9) Thread[id=16, name=h2sc-2-thread-1, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=19, name=qtp2107975691-19, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=17, name=h2sc-2-thread-2, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=20, name=qtp2107975691-20-acceptor-0@21dc258-ServerConnector@1fcdb038{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33009}, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=24, name=qtp2107975691-24, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=27, name=Session-HouseKeeper-588b38bb-1, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=39, name=searcherExecutor-14-thread-1, state=WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=26, name=qtp2107975691-26, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   17) Thread[id=23, name=qtp2107975691-23, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   18) Thread[id=32, name=h2sc-9-thread-1, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$229/0x00000008402d9c40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$230/0x00000008402d9440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> juil. 09, 2023 6:29:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFOS: Starting to interrupt leaked threads:
  2>    1) Thread[id=28, name=httpShardExecutor-7-thread-1, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>    2) Thread[id=21, name=qtp2107975691-21, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>    3) Thread[id=33, name=h2sc-9-thread-2, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>    4) Thread[id=15, name=TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14], state=WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>    5) Thread[id=22, name=qtp2107975691-22, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>    6) Thread[id=25, name=qtp2107975691-25, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>    7) Thread[id=29, name=httpShardExecutor-7-thread-2, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>    8) Thread[id=35, name=CloserThread, state=WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>    9) Thread[id=16, name=h2sc-2-thread-1, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>   10) Thread[id=19, name=qtp2107975691-19, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>   11) Thread[id=17, name=h2sc-2-thread-2, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>   12) Thread[id=20, name=qtp2107975691-20-acceptor-0@21dc258-ServerConnector@1fcdb038{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:33009}, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2>   13) Thread[id=24, name=qtp2107975691-24, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>   14) Thread[id=27, name=Session-HouseKeeper-588b38bb-1, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>   15) Thread[id=39, name=searcherExecutor-14-thread-1, state=WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>   16) Thread[id=26, name=qtp2107975691-26, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>   17) Thread[id=23, name=qtp2107975691-23, state=TIMED_WAITING, group=TGRP-TestLBHttp2SolrClient]
  2>   18) Thread[id=32, name=h2sc-9-thread-1, state=RUNNABLE, group=TGRP-TestLBHttp2SolrClient]
  2> 7203918 ERROR (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter.
  2>           => org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:554)
  2> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:554) ~[solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]]
  2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT d6d8d65eceeccdb97159daab8d392fd06999aae3 [snapshot build, details omitted]]
  2> 	at org.apache.solr.client.solrj.TestLBHttp2SolrClient$SolrInstance.startJetty(TestLBHttp2S

[...truncated too long message...]

- 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7204419 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.Server Stopped Server@a77751{STOPPING}[10.0.15,sto=0]
  2> 7204423 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.AbstractConnector Stopped ServerConnector@1fcdb038{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 7204425 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6f47bc64{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 7204433 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=809278668
  2> 7204435 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@1ad0fb39 collection1
  2> 7204435 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@1ad0fb39
  2> 7204462 INFO  (coreCloseExecutor-19-thread-1) [ x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 7204483 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7204492 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7204496 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7204512 INFO  (TEST-TestLBHttp2SolrClient.testTwoServers-seed#[A62F678E68998A14]) [] o.a.s.SolrTestCaseJ4 ###Ending testTwoServers
  2> 7204515 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 7204516 INFO  (SUITE-TestLBHttp2SolrClient-seed#[A62F678E68998A14]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttp2SolrClient_A62F678E68998A14-001
  2> Jul 10, 2023 1:29:35 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([A62F678E68998A14]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=939, maxMBSortInHeap=5.471435602162218, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=fr-GF, timezone=America/Hermosillo
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=267209768,total=324534272
  2> NOTE: All tests run in this JVM: [TestLBHttp2SolrClient]
  2> NOTE: reproduce with: gradlew test --tests TestLBHttp2SolrClient -Dtests.seed=A62F678E68998A14 -Dtests.locale=fr-GF -Dtests.timezone=America/Hermosillo -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:solrj:test (FAILURE): 1133 test(s), 2 failure(s), 40 skipped

1133 tests completed, 2 failed, 40 skipped

> Task :solr:solrj:test FAILED
> Task :solr:solrj:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttp2SolrClient.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.TestLBHttp2SolrClient.testTwoServers" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A62F678E68998A14 -Ptests.file.encoding=US-ASCII

  - org.apache.solr.client.solrj.TestLBHttp2SolrClient.classMethod (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttp2SolrClient.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.TestLBHttp2SolrClient.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A62F678E68998A14 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:solrj:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/solrj/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 29s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m51s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4841 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4841/

All tests passed

Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22599ms
ok 43 post solr log into solr in 9386ms
ok 44 start solr with ssl in 19653ms
ok 45 start solr with ssl and auth in 22274ms
ok 46 SOLR-11740 check 'solr stop' connection in 28145ms
ok 47 status detects locally running solr in 14478ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2213ms
ok 50 copying files around in 6524ms
ok 51 upconfig in 2641ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m23.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4840 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4840/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22523ms
ok 43 post solr log into solr in 8775ms
ok 44 start solr with ssl in 19918ms
ok 45 start solr with ssl and auth in 22629ms
ok 46 SOLR-11740 check 'solr stop' connection in 28360ms
ok 47 status detects locally running solr in 14355ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2143ms
ok 50 copying files around in 6487ms
ok 51 upconfig in 2609ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m23.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4839 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4839/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22398ms
ok 43 post solr log into solr in 8889ms
ok 44 start solr with ssl in 19995ms
ok 45 start solr with ssl and auth in 22341ms
ok 46 SOLR-11740 check 'solr stop' connection in 28107ms
ok 47 status detects locally running solr in 14379ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2179ms
ok 50 copying files around in 6521ms
ok 51 upconfig in 2636ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 9s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m31.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4838 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4838/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22743ms
ok 43 post solr log into solr in 9154ms
ok 44 start solr with ssl in 20133ms
ok 45 start solr with ssl and auth in 22412ms
ok 46 SOLR-11740 check 'solr stop' connection in 28146ms
ok 47 status detects locally running solr in 14418ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2179ms
ok 50 copying files around in 6647ms
ok 51 upconfig in 2652ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 11s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m34.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4837 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4837/

2 tests failed.
FAILED:  org.apache.solr.handler.designer.TestSchemaDesignerAPI.classMethod

Error Message:
java.util.concurrent.TimeoutException: Timed out waiting for all collections to be fully removed.

Stack Trace:
java.util.concurrent.TimeoutException: Timed out waiting for all collections to be fully removed.
	at __randomizedtesting.SeedInfo.seed([4249847E1D7D596]:0)
	at org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:678)
	at org.apache.solr.handler.designer.TestSchemaDesignerAPI.tearDownCluster(TestSchemaDesignerAPI.java:81)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:909)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.handler.designer.TestSchemaDesignerAPI.classMethod

Error Message:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=410, name=coreCloseExecutor-437-thread-1, state=RUNNABLE, group=TGRP-TestSchemaDesignerAPI]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=410, name=coreCloseExecutor-437-thread-1, state=RUNNABLE, group=TGRP-TestSchemaDesignerAPI]
Caused by: java.lang.AssertionError: Too many closes on SolrCore
	at __randomizedtesting.SeedInfo.seed([4249847E1D7D596]:0)
	at org.apache.solr.core.SolrCore.close(SolrCore.java:1757)
	at org.apache.solr.core.SolrCores.lambda$close$0(SolrCores.java:128)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 7009 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22350ms
ok 43 post solr log into solr in 9170ms
ok 44 start solr with ssl in 19939ms
ok 45 start solr with ssl and auth in 22984ms
ok 46 SOLR-11740 check 'solr stop' connection in 28122ms
ok 47 status detects locally running solr in 14368ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2162ms
ok 50 copying files around in 6521ms
ok 51 upconfig in 2616ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.handler.designer.TestSchemaDesignerAPI.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.handler.designer.TestSchemaDesignerAPI.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.handler.designer.TestSchemaDesignerAPI.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=4249847E1D7D596 -Ptests.file.encoding=UTF-8

  - org.apache.solr.handler.designer.TestSchemaDesignerAPI.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.handler.designer.TestSchemaDesignerAPI.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.handler.designer.TestSchemaDesignerAPI.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=4249847E1D7D596 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 3s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4836 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4836/

All tests passed

Build Log:
[...truncated 1249 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22487ms
ok 43 post solr log into solr in 9047ms
ok 44 start solr with ssl in 19910ms
ok 45 start solr with ssl and auth in 22685ms
ok 46 SOLR-11740 check 'solr stop' connection in 28122ms
ok 47 status detects locally running solr in 14723ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2179ms
ok 50 copying files around in 6565ms
ok 51 upconfig in 2617ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m22s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4835 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4835/

All tests passed

Build Log:
[...truncated 1263 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22349ms
ok 43 post solr log into solr in 8945ms
ok 44 start solr with ssl in 20674ms
ok 45 start solr with ssl and auth in 22756ms
ok 46 SOLR-11740 check 'solr stop' connection in 28122ms
ok 47 status detects locally running solr in 14399ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6503ms
ok 51 upconfig in 2639ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 2s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4834 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4834/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23001ms
ok 43 post solr log into solr in 9113ms
ok 44 start solr with ssl in 20206ms
ok 45 start solr with ssl and auth in 22395ms
ok 46 SOLR-11740 check 'solr stop' connection in 28195ms
ok 47 status detects locally running solr in 14419ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2221ms
ok 50 copying files around in 6622ms
ok 51 upconfig in 2677ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 59s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4833 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4833/

All tests passed

Build Log:
[...truncated 1252 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22087ms
ok 43 post solr log into solr in 8896ms
ok 44 start solr with ssl in 20127ms
ok 45 start solr with ssl and auth in 22625ms
ok 46 SOLR-11740 check 'solr stop' connection in 28195ms
ok 47 status detects locally running solr in 14351ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2161ms
ok 50 copying files around in 6459ms
ok 51 upconfig in 2607ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 51s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m12.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4832 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4832/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22474ms
ok 43 post solr log into solr in 9080ms
ok 44 start solr with ssl in 19710ms
ok 45 start solr with ssl and auth in 22490ms
ok 46 SOLR-11740 check 'solr stop' connection in 28141ms
ok 47 status detects locally running solr in 14389ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2182ms
ok 50 copying files around in 6510ms
ok 51 upconfig in 2618ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 56s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m20s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4831 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4831/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22168ms
ok 43 post solr log into solr in 8784ms
ok 44 start solr with ssl in 19567ms
ok 45 start solr with ssl and auth in 22241ms
ok 46 SOLR-11740 check 'solr stop' connection in 28372ms
ok 47 status detects locally running solr in 14328ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2152ms
ok 50 copying files around in 6446ms
ok 51 upconfig in 2611ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 52s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m14.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4830 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4830/

All tests passed

Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22105ms
ok 43 post solr log into solr in 8844ms
ok 44 start solr with ssl in 19973ms
ok 45 start solr with ssl and auth in 22126ms
ok 46 SOLR-11740 check 'solr stop' connection in 28361ms
ok 47 status detects locally running solr in 14609ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2141ms
ok 50 copying files around in 6469ms
ok 51 upconfig in 2595ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 14m 7s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 14m28.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4829 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4829/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23604ms
ok 43 post solr log into solr in 9704ms
ok 44 start solr with ssl in 20521ms
ok 45 start solr with ssl and auth in 23002ms
ok 46 SOLR-11740 check 'solr stop' connection in 28375ms
ok 47 status detects locally running solr in 14537ms
ok 48 status does not expose cli parameters to end user in 232ms
ok 49 listing out files in 2213ms
ok 50 copying files around in 6746ms
ok 51 upconfig in 2821ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 32s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m58.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4828 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4828/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22632ms
ok 43 post solr log into solr in 8975ms
ok 44 start solr with ssl in 20286ms
ok 45 start solr with ssl and auth in 22376ms
ok 46 SOLR-11740 check 'solr stop' connection in 28510ms
ok 47 status detects locally running solr in 14391ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2150ms
ok 50 copying files around in 6562ms
ok 51 upconfig in 2614ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 59s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m21s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4827 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4827/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22771ms
ok 43 post solr log into solr in 9042ms
ok 44 start solr with ssl in 19740ms
ok 45 start solr with ssl and auth in 22534ms
ok 46 SOLR-11740 check 'solr stop' connection in 28460ms
ok 47 status detects locally running solr in 14725ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2156ms
ok 50 copying files around in 6481ms
ok 51 upconfig in 2611ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 5s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m27s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4826 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4826/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22342ms
ok 43 post solr log into solr in 9336ms
ok 44 start solr with ssl in 20040ms
ok 45 start solr with ssl and auth in 22586ms
ok 46 SOLR-11740 check 'solr stop' connection in 28168ms
ok 47 status detects locally running solr in 14395ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2162ms
ok 50 copying files around in 6581ms
ok 51 upconfig in 2626ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m22.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4825 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4825/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22642ms
ok 43 post solr log into solr in 9016ms
ok 44 start solr with ssl in 19729ms
ok 45 start solr with ssl and auth in 22193ms
ok 46 SOLR-11740 check 'solr stop' connection in 28329ms
ok 47 status detects locally running solr in 14375ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2182ms
ok 50 copying files around in 6534ms
ok 51 upconfig in 2644ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 52s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m13.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4824 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4824/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22322ms
ok 43 post solr log into solr in 8981ms
ok 44 start solr with ssl in 20038ms
ok 45 start solr with ssl and auth in 22316ms
ok 46 SOLR-11740 check 'solr stop' connection in 28280ms
ok 47 status detects locally running solr in 14358ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2162ms
ok 50 copying files around in 6626ms
ok 51 upconfig in 2640ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 3s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m24.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4823 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4823/

1 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTest.testShardLeaderChange

Error Message:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=405, name=OverseerCollectionConfigSetProcessor-72057616714694685-127.0.0.1:36495_solr-n_0000000000, state=RUNNABLE, group=Overseer collection creation process.]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught exception in thread: Thread[id=405, name=OverseerCollectionConfigSetProcessor-72057616714694685-127.0.0.1:36495_solr-n_0000000000, state=RUNNABLE, group=Overseer collection creation process.]
	at __randomizedtesting.SeedInfo.seed([4A9B91B60DDA1CBC:94C816411742E94D]:0)
Caused by: org.apache.solr.common.AlreadyClosedException
	at __randomizedtesting.SeedInfo.seed([4A9B91B60DDA1CBC]:0)
	at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70)
	at app//org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:437)
	at app//org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:486)
	at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:171)
	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 16169 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22493ms
ok 43 post solr log into solr in 9034ms
ok 44 start solr with ssl in 19726ms
not ok 45 start solr with ssl and auth in 1405907ms
# (in test file test/test_ssl.bats, line 87)
#   `solr auth enable -type basicAuth -credentials name:password' failed
# NOTE: Please install lsof as this script needs it to determine if Solr is listening on port 8983.
#
# Started Solr server on port 8983 (pid=180525). Happy searching!
#
# INFO  - 2023-07-08 01:56:57.251; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO  - 2023-07-08 01:56:57.257; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# Couldn't get ZooKeeper host. Please make sure Solr is running in cloud mode, or a zkHost has been passed in.
# Please find the SOLR_HOME snapshot for failed test #2 at: /tmp/src/solr/solr/packaging/build/test-output/failure-snapshots/45-2
# Last output:
# INFO  - 2023-07-08 02:06:58.831; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.keyStorePassword
# INFO  - 2023-07-08 02:06:58.836; org.apache.solr.util.configuration.SSLConfigurations; Setting javax.net.ssl.trustStorePassword
# Exception in thread "main" org.apache.solr.client.solrj.SolrServerException: Timeout occurred while waiting response from server at: https://localhost:8983/solr/admin/info/system?wt=javabin&version=2
# 	at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:522)
# 	at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1192)
# 	at org.apache.solr.cli.SolrCLI.getZkHost(SolrCLI.java:559)
# 	at org.apache.solr.cli.AuthTool.clearSecurityJson(AuthTool.java:483)
# 	at org.apache.solr.cli.AuthTool.handleBasicAuth(AuthTool.java:440)
# 	at org.apache.solr.cli.AuthTool.runTool(AuthTool.java:151)
# 	at org.apache.solr.cli.SolrCLI.main(SolrCLI.java:156)
# Caused by: java.util.concurrent.TimeoutException
# 	at org.eclipse.jetty.client.util.InputStreamResponseListener.get(InputStreamResponseListener.java:214)
# 	at org.apache.solr.client.solrj.impl.Http2SolrClient.request(Http2SolrClient.java:512)
# 	... 6 more
ok 46 SOLR-11740 check 'solr stop' connection in 28169ms
ok 47 status detects locally running solr in 14411ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2176ms
ok 50 copying files around in 6585ms
ok 51 upconfig in 2678ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.OverseerTest.testShardLeaderChange (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.OverseerTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.OverseerTest.testShardLeaderChange" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=4A9B91B60DDA1CBC -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 39m 2s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 39m32.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4822 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4822/

2 tests failed.
FAILED:  org.apache.solr.ltr.TestLTRWithFacet.testRankingSolrFacet

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([B3198640AD269FDE]:0)


FAILED:  org.apache.solr.ltr.TestLTRWithFacet.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([B3198640AD269FDE]:0)




Build Log:
[...truncated 1237 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22245ms
ok 43 post solr log into solr in 8857ms
ok 44 start solr with ssl in 19702ms
ok 45 start solr with ssl and auth in 22211ms
ok 46 SOLR-11740 check 'solr stop' connection in 28290ms
ok 47 status detects locally running solr in 14389ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2154ms
ok 50 copying files around in 6519ms
ok 51 upconfig in 2608ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.TestLTRWithFacet > testRankingSolrFacet FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([B3198640AD269FDE]:0)

org.apache.solr.ltr.TestLTRWithFacet > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([B3198640AD269FDE]:0)

org.apache.solr.ltr.TestLTRWithFacet > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTRWithFacet.txt, copied below:
  1> 22:01:09.381 [TEST-TestLTRWithFacet.testRankingSolrFacet-seed#[B3198640AD269FDE]] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> јул 08, 2023 3:01:08 ПРИЈЕ ПОДНЕ com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.ltr.TestLTRWithFacet
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@5ddb5620
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@5ddb5620
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6623ac11
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@6623ac11
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@7fbe01a8
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@7fbe01a8
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:40960 to /127.0.0.1:34893 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4c5864e0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4c5864e0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:40960 to /127.0.0.1:34893 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ef5a3d8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3ef5a3d8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:40960 to /127.0.0.1:34893 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@454dd333
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4a712973
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestLTRWithFacet-seed#[B3198640AD269FDE]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@45012bb9
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestLTRWithFacet.testRankingSolrFacet-seed#[B3198640AD269FDE]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@25ea6e8f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@25ea6e8f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:508)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:547)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@56f9bba2
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@7d063e05
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.TestLTRWithFacet.before(TestLTRWithFacet.java:31)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@3e7faa67
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@72ef02f2
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@69d083a6
  2> 
  2> "qtp845619600-17-acceptor-0@69dfbc7d-ServerConnector@26966e51{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37889}" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@6685c4c7
  2> 
  2> "qtp845619600-18" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6c77b277
  2> 	- locked sun.nio.ch.EPollSelectorImpl@157d1ec
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp845619600-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp845619600-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp845619600-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp845619600-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp845619600-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp845619600-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@773f595f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-792fc7ab-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f541883
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f541883
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6ee1f114
  2> 	- locked sun.nio.ch.EPollSelectorImpl@51869595
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@7b8548ee
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5c7edb5b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@43715057
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@9b1aa06
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2279607c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@47b6739b
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@16d3e1f0
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6d38cc35
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2402ca2d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@75bd840b
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@4d913451
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@4d913451
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@39cfacd3
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@39cfacd3
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12bd86bb
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@12bd86bb
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([B3198640AD269FDE]:0)
  2> NOTE: reproduce with: gradlew test --tests TestLTRWithFacet.testRankingSolrFacet -Dtests.seed=B3198640AD269FDE -Dtests.locale=sr-Cyrl-BA -Dtests.timezone=Africa/Addis_Ababa -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> јул 08, 2023 3:01:08 ПРИЈЕ ПОДНЕ com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> јул 08, 2023 3:01:09 ПРИЈЕ ПОДНЕ com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.TestLTRWithFacet: 
  2>    1) Thread[id=25, name=Session-HouseKeeper-792fc7ab-1, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=17, name=qtp845619600-17-acceptor-0@69dfbc7d-ServerConnector@26966e51{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37889}, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=15, name=TEST-TestLTRWithFacet.testRankingSolrFacet-seed#[B3198640AD269FDE], state=WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:508)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:547)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.TestLTRWithFacet.before(TestLTRWithFacet.java:31)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=23, name=qtp845619600-23, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=24, name=qtp845619600-24, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   10) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=20, name=qtp845619600-20, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=22, name=qtp845619600-22, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=21, name=qtp845619600-21, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=18, name=qtp845619600-18, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=19, name=qtp845619600-19, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> јул 08, 2023 3:01:09 ПРИЈЕ ПОДНЕ com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=25, name=Session-HouseKeeper-792fc7ab-1, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>    2) Thread[id=17, name=qtp845619600-17-acceptor-0@69dfbc7d-ServerConnector@26966e51{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37889}, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>    3) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>    4) Thread[id=15, name=TEST-TestLTRWithFacet.testRankingSolrFacet-seed#[B3198640AD269FDE], state=WAITING, group=TGRP-TestLTRWithFacet]
  2>    5) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>    6) Thread[id=23, name=qtp845619600-23, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>    7) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestLTRWithFacet]
  2>    8) Thread[id=24, name=qtp845619600-24, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>    9) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestLTRWithFacet]
  2>   10) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>   11) Thread[id=20, name=qtp845619600-20, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>   12) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>   13) Thread[id=22, name=qtp845619600-22, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>   14) Thread[id=21, name=qtp845619600-21, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  2>   15) Thread[id=18, name=qtp845619600-18, state=RUNNABLE, group=TGRP-TestLTRWithFacet]
  2>   16) Thread[id=19, name=qtp845619600-19, state=TIMED_WAITING, group=TGRP-TestLTRWithFacet]
  1> 00:01:09.275 [TEST-TestLTRWithFacet.testRankingSolrFacet-seed#[B3198640AD269FDE]] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:554) ~[solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.4.0-SNAPSHOT.jar:9.4.0-SNAPSHOT 6212f87d1ba0b86c898cca1c62d8699360bd94ff [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.TestLTRWithFacet.before(TestLTRWithFacet.java:31) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.7.0.jar:9.7.0 ccf4b198ec328095d45d2746189dc8ca633e8bcf - 2023-06-21 11:48:16]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/modules/ltr/build/tmp/tests-tmp/solr.ltr.TestLTRWithFacet_B3198640AD269FDE-001
  2> Jul 08, 2023 12:01:10 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([B3198640AD269FDE]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=408, maxMBSortInHeap=5.937988715835267, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=sr-Cyrl-BA, timezone=Africa/Addis_Ababa
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=249960568,total=324534272
  2> NOTE: All tests run in this JVM: [TestLTRWithFacet]
  2> NOTE: reproduce with: gradlew test --tests TestLTRWithFacet -Dtests.seed=B3198640AD269FDE -Dtests.locale=sr-Cyrl-BA -Dtests.timezone=Africa/Addis_Ababa -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 187 test(s), 2 failure(s)

187 tests completed, 2 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.TestLTRWithFacet.testRankingSolrFacet (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTRWithFacet.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.TestLTRWithFacet.testRankingSolrFacet" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B3198640AD269FDE -Ptests.file.encoding=US-ASCII

  - org.apache.solr.ltr.TestLTRWithFacet.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTRWithFacet.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.TestLTRWithFacet.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B3198640AD269FDE -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 45s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h3m9s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4821 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4821/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22399ms
ok 43 post solr log into solr in 9085ms
ok 44 start solr with ssl in 20130ms
ok 45 start solr with ssl and auth in 22440ms
ok 46 SOLR-11740 check 'solr stop' connection in 28185ms
ok 47 status detects locally running solr in 14406ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2186ms
ok 50 copying files around in 6550ms
ok 51 upconfig in 2666ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 8s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m31s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4820 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4820/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22444ms
ok 43 post solr log into solr in 9081ms
ok 44 start solr with ssl in 20228ms
ok 45 start solr with ssl and auth in 22386ms
ok 46 SOLR-11740 check 'solr stop' connection in 28115ms
ok 47 status detects locally running solr in 14694ms
ok 48 status does not expose cli parameters to end user in 217ms
ok 49 listing out files in 2167ms
ok 50 copying files around in 6563ms
ok 51 upconfig in 2630ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 11s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m32.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4819 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4819/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22726ms
ok 43 post solr log into solr in 9202ms
ok 44 start solr with ssl in 19676ms
ok 45 start solr with ssl and auth in 22896ms
ok 46 SOLR-11740 check 'solr stop' connection in 28418ms
ok 47 status detects locally running solr in 14344ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2147ms
ok 50 copying files around in 6554ms
ok 51 upconfig in 2632ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 13s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m35.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4818 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4818/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22515ms
ok 43 post solr log into solr in 9111ms
ok 44 start solr with ssl in 20333ms
ok 45 start solr with ssl and auth in 22500ms
ok 46 SOLR-11740 check 'solr stop' connection in 28138ms
ok 47 status detects locally running solr in 14462ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2177ms
ok 50 copying files around in 6612ms
ok 51 upconfig in 2634ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 12s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m35s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4817 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4817/

1 tests failed.
FAILED:  org.apache.solr.cloud.ReindexCollectionTest.testAbort

Error Message:
java.lang.AssertionError: {responseHeader={status=0, QTime=6}, status={state=running, msg=found [d1ffa352-cda7-47eb-a569-214f81c5554d] in running tasks}}

Stack Trace:
java.lang.AssertionError: {responseHeader={status=0, QTime=6}, status={state=running, msg=found [d1ffa352-cda7-47eb-a569-214f81c5554d] in running tasks}}
	at __randomizedtesting.SeedInfo.seed([AA164F808E5349A0:CC75F817FA89DBD9]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertNotNull(Assert.java:713)
	at org.apache.solr.cloud.ReindexCollectionTest.testAbort(ReindexCollectionTest.java:425)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 6665 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22546ms
ok 43 post solr log into solr in 9069ms
ok 44 start solr with ssl in 20001ms
ok 45 start solr with ssl and auth in 22382ms
ok 46 SOLR-11740 check 'solr stop' connection in 28065ms
ok 47 status detects locally running solr in 14659ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2179ms
ok 50 copying files around in 6559ms
ok 51 upconfig in 2638ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.ReindexCollectionTest.testAbort (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ReindexCollectionTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.ReindexCollectionTest.testAbort" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=AA164F808E5349A0 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 13s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m37.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4816 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4816/

1 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testNRTRestart

Error Message:
java.lang.AssertionError: initial error time threshold exceeded

Stack Trace:
java.lang.AssertionError: initial error time threshold exceeded
	at __randomizedtesting.SeedInfo.seed([FB4F3D9F56B81D51:BDB8D54F4A5CF87]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.solr.search.TestCoordinatorRole.getHostCoreName(TestCoordinatorRole.java:474)
	at org.apache.solr.search.TestCoordinatorRole.testNRTRestart(TestCoordinatorRole.java:399)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 6931 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22270ms
ok 43 post solr log into solr in 9588ms
ok 44 start solr with ssl in 19927ms
ok 45 start solr with ssl and auth in 22172ms
ok 46 SOLR-11740 check 'solr stop' connection in 28422ms
ok 47 status detects locally running solr in 14364ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2166ms
ok 50 copying files around in 6584ms
ok 51 upconfig in 2666ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testNRTRestart (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testNRTRestart" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=FB4F3D9F56B81D51 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 5s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m27s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4815 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4815/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22593ms
ok 43 post solr log into solr in 8920ms
ok 44 start solr with ssl in 20141ms
ok 45 start solr with ssl and auth in 22859ms
ok 46 SOLR-11740 check 'solr stop' connection in 28350ms
ok 47 status detects locally running solr in 14412ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2183ms
ok 50 copying files around in 6524ms
ok 51 upconfig in 2660ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m24s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4814 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4814/

All tests passed

Build Log:
[...truncated 1246 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22819ms
ok 43 post solr log into solr in 8703ms
ok 44 start solr with ssl in 20611ms
ok 45 start solr with ssl and auth in 22368ms
ok 46 SOLR-11740 check 'solr stop' connection in 28328ms
ok 47 status detects locally running solr in 14365ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2152ms
ok 50 copying files around in 6462ms
ok 51 upconfig in 2588ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 4s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4813 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4813/

All tests passed

Build Log:
[...truncated 1252 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22370ms
ok 43 post solr log into solr in 8893ms
ok 44 start solr with ssl in 20455ms
ok 45 start solr with ssl and auth in 22993ms
ok 46 SOLR-11740 check 'solr stop' connection in 28159ms
ok 47 status detects locally running solr in 14386ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2148ms
ok 50 copying files around in 6514ms
ok 51 upconfig in 2628ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 5s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m30.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4812 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4812/

All tests passed

Build Log:
[...truncated 1246 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22271ms
ok 43 post solr log into solr in 9392ms
ok 44 start solr with ssl in 20424ms
ok 45 start solr with ssl and auth in 22861ms
ok 46 SOLR-11740 check 'solr stop' connection in 28096ms
ok 47 status detects locally running solr in 14341ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2168ms
ok 50 copying files around in 6581ms
ok 51 upconfig in 2645ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 9s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m31.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4811 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4811/

3 tests failed.
FAILED:  org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.mockOAuth2Server

Error Message:
org.junit.ComparisonFailure: Should have received 401 code expected:<[401]> but was:<[200]>

Stack Trace:
org.junit.ComparisonFailure: Should have received 401 code expected:<[401]> but was:<[200]>
	at __randomizedtesting.SeedInfo.seed([B85C0E96A901D646:3F5CEFC6CCBD2AAA]:0)
	at org.junit.Assert.assertEquals(Assert.java:117)
	at org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.mockOAuth2Server(JWTAuthPluginIntegrationTest.java:137)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 63 threads leaked from SUITE scope at org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest: 
   1) Thread[id=299, name=zkConnectionManagerCallback-165-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=261, name=Session-HouseKeeper-1b2ffc5b-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=283, name=jetty-launcher-125-thread-1-EventThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
   4) Thread[id=248, name=qtp1877936586-248-acceptor-0@4263ab5e-ServerConnector@6128ea47{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:34405}, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=260, name=qtp1274752886-260, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=227, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
   7) Thread[id=238, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
   8) Thread[id=279, name=h2sc-143-thread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=286, name=zkConnectionManagerCallback-147-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=291, name=OverseerCollectionConfigSetProcessor-72057945644007429-127.0.0.1:36117_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
        at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
        at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  11) Thread[id=288, name=zkCallback-146-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  12) Thread[id=246, name=qtp1877936586-246, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  13) Thread[id=229, name=SyncThread:0, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  14) Thread[id=272, name=httpShardExecutor-139-thread-2, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  15) Thread[id=225, name=NIOServerCxnFactory.SelectorThread-0, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  16) Thread[id=284, name=jetty-launcher-125-thread-2-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  17) Thread[id=304, name=zkCallback-148-thread-2, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  18) Thread[id=235, name=NIOWorkerThread-2, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  19) Thread[id=289, name=zkCallback-148-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  20) Thread[id=249, name=qtp1274752886-249-acceptor-0@5742e6ba-ServerConnector@453cc879{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:36117}, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  21) Thread[id=263, name=Session-HouseKeeper-3e34705b-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  22) Thread[id=281, name=h2sc-143-thread-2, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  23) Thread[id=270, name=httpShardExecutor-139-thread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  24) Thread[id=273, name=httpShardExecutor-138-thread-2, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  25) Thread[id=271, name=httpShardExecutor-138-thread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  26) Thread[id=237, name=zkConnectionManagerCallback-120-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  27) Thread[id=297, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  28) Thread[id=232, name=NIOWorkerThread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  29) Thread[id=256, name=qtp1877936586-256, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  30) Thread[id=253, name=qtp1877936586-253, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  31) Thread[id=226, name=NIOServerCxnFactory.SelectorThread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  32) Thread[id=233, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  33) Thread[id=240, name=zkConnectionManagerCallback-122-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  34) Thread[id=255, name=qtp1274752886-255, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  35) Thread[id=287, name=zkConnectionManagerCallback-149-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  36) Thread[id=294, name=CloserThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/java.lang.Object.wait(Native Method)
        at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  37) Thread[id=254, name=qtp1877936586-254, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  38) Thread[id=258, name=qtp1877936586-258, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  39) Thread[id=257, name=qtp1274752886-257, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  40) Thread[id=230, name=ProcessThread(sid:0 cport:46089):, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  41) Thread[id=231, name=RequestThrottler, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  42) Thread[id=251, name=qtp1877936586-251, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  43) Thread[id=285, name=jetty-launcher-125-thread-2-EventThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  44) Thread[id=293, name=coreContainerWorkExecutor-131-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  45) Thread[id=290, name=OverseerStateUpdate-72057945644007429-127.0.0.1:36117_solr-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
        at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  46) Thread[id=282, name=jetty-launcher-125-thread-1-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  47) Thread[id=239, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-EventThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  48) Thread[id=308, name=Connector-Scheduler-453cc879-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  49) Thread[id=223, name=ZkTestServer Run Thread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/java.lang.Object.wait(Native Method)
        at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
        at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:915)
        at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
        at app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  50) Thread[id=262, name=qtp1274752886-262, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  51) Thread[id=295, name=coreContainerWorkExecutor-134-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  52) Thread[id=278, name=h2sc-142-thread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  53) Thread[id=292, name=CloserThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/java.lang.Object.wait(Native Method)
        at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
        at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  54) Thread[id=280, name=h2sc-142-thread-2, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  55) Thread[id=259, name=qtp1274752886-259, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  56) Thread[id=247, name=qtp1274752886-247, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  57) Thread[id=303, name=zkCallback-164-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  58) Thread[id=234, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-EventThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  59) Thread[id=298, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-EventThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  60) Thread[id=305, name=zkCallback-146-thread-2, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  61) Thread[id=236, name=NIOWorkerThread-3, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  62) Thread[id=252, name=qtp1274752886-252, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  63) Thread[id=250, name=qtp1877936586-250, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 63 threads leaked from SUITE scope at org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest: 
   1) Thread[id=299, name=zkConnectionManagerCallback-165-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   2) Thread[id=261, name=Session-HouseKeeper-1b2ffc5b-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   3) Thread[id=283, name=jetty-launcher-125-thread-1-EventThread, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
   4) Thread[id=248, name=qtp1877936586-248-acceptor-0@4263ab5e-ServerConnector@6128ea47{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:34405}, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   5) Thread[id=260, name=qtp1274752886-260, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   6) Thread[id=227, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
   7) Thread[id=238, name=TEST-JWTAuthPluginIntegrationTest.mockOAuth2Server-seed#[B85C0E96A901D646]-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
   8) Thread[id=279, name=h2sc-143-thread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   9) Thread[id=286, name=zkConnectionManagerCallback-147-thread-1, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  10) Thread[id=291, name=OverseerCollectionConfigSetProcessor-72057945644007429-127.0.0.1:36117_solr-n_0000000000, state=TIMED_WAITING, group=Overseer collection creation process.]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
        at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
        at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  11) Thread[id=288, name=zkCallback-146-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  12) Thread[id=246, name=qtp1877936586-246, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  13) Thread[id=229, name=SyncThread:0, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  14) Thread[id=272, name=httpShardExecutor-139-thread-2, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  15) Thread[id=225, name=NIOServerCxnFactory.SelectorThread-0, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
        at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  16) Thread[id=284, name=jetty-launcher-125-thread-2-SendThread(127.0.0.1:46089), state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
        at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
        at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  17) Thread[id=304, name=zkCallback-148-thread-2, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  18) Thread[id=235, name=NIOWorkerThread-2, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  19) Thread[id=289, name=zkCallback-148-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  20) Thread[id=249, name=qtp1274752886-249-acceptor-0@5742e6ba-ServerConnector@453cc879{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:36117}, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
        at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
        at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
        at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  21) Thread[id=263, name=Session-HouseKeeper-3e34705b-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
        at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  22) Thread[id=281, name=h2sc-143-thread-2, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
        at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$262/0x000000084041fc40.run(Unknown Source)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  23) Thread[id=270, name=httpShardExecutor-139-thread-1, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.

[...truncated too long message...]

ase@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  47) Thread[id=247, name=qtp1274752886-247, state=RUNNABLE, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
        at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
        at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
        at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
        at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
        at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
        at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
        at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$323/0x0000000840485440.run(Unknown Source)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  48) Thread[id=303, name=zkCallback-164-thread-1, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  49) Thread[id=305, name=zkCallback-146-thread-2, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
        at java.base@11.0.16/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  50) Thread[id=236, name=NIOWorkerThread-3, state=WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
        at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
        at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  51) Thread[id=252, name=qtp1274752886-252, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  52) Thread[id=250, name=qtp1877936586-250, state=TIMED_WAITING, group=TGRP-JWTAuthPluginIntegrationTest]
        at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
        at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([B85C0E96A901D646]:0)




Build Log:
[...truncated 4994 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.mockOAuth2Server (:solr:modules:jwt-auth)
    Test output: /tmp/src/solr/solr/modules/jwt-auth/build/test-results/test/outputs/OUTPUT-org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.txt
    Reproduce with: gradlew :solr:modules:jwt-auth:test --tests "org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.mockOAuth2Server" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B85C0E96A901D646 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.classMethod (:solr:modules:jwt-auth)
    Test output: /tmp/src/solr/solr/modules/jwt-auth/build/test-results/test/outputs/OUTPUT-org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.txt
    Reproduce with: gradlew :solr:modules:jwt-auth:test --tests "org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B85C0E96A901D646 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.classMethod (:solr:modules:jwt-auth)
    Test output: /tmp/src/solr/solr/modules/jwt-auth/build/test-results/test/outputs/OUTPUT-org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.txt
    Reproduce with: gradlew :solr:modules:jwt-auth:test --tests "org.apache.solr.security.jwt.JWTAuthPluginIntegrationTest.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B85C0E96A901D646 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':solr:modules:jwt-auth:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/jwt-auth/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 5m 15s
616 actionable tasks: 600 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 5m37s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4810 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4810/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22431ms
ok 43 post solr log into solr in 9004ms
ok 44 start solr with ssl in 19889ms
ok 45 start solr with ssl and auth in 22700ms
ok 46 SOLR-11740 check 'solr stop' connection in 28371ms
ok 47 status detects locally running solr in 14625ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6551ms
ok 51 upconfig in 2615ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m22.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4809 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4809/

2 tests failed.
FAILED:  org.apache.solr.client.solrj.request.SchemaTest.testCopyFieldAccuracy

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([93453D83EA81F2EB]:0)


FAILED:  org.apache.solr.client.solrj.request.SchemaTest.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([93453D83EA81F2EB]:0)




Build Log:
[...truncated 1244 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22970ms
ok 43 post solr log into solr in 9166ms
ok 44 start solr with ssl in 20547ms
ok 45 start solr with ssl and auth in 22885ms
ok 46 SOLR-11740 check 'solr stop' connection in 28132ms
ok 47 status detects locally running solr in 14387ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2196ms
ok 50 copying files around in 6571ms
ok 51 upconfig in 2657ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:solrj:test

org.apache.solr.client.solrj.request.SchemaTest > testCopyFieldAccuracy FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([93453D83EA81F2EB]:0)

org.apache.solr.client.solrj.request.SchemaTest > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([93453D83EA81F2EB]:0)

org.apache.solr.client.solrj.request.SchemaTest > test suite's output saved to /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.request.SchemaTest.txt, copied below:
  2> 3479 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 3501 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 3540 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 3586 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/data-dir-1-001
  2> 3701 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 3738 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 3807 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.SolrTestCaseJ4 ###Starting testCopyFieldAccuracy
  2> 3892 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.SolrTestCaseJ4 ####initCore
  2> 3893 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.SolrTestCaseJ4 ####initCore end
  2> 3895 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.SolrTestCaseJ4 Writing core.properties file to /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-003/cores/core
  2> 4410 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 4413 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 4423 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 4584 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 4634 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@406992ec{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 4673 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.AbstractConnector Started ServerConnector@10e4fe95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42931}
  2> 4685 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.Server Started Server@54330f13{STARTING}[10.0.15,sto=0] @4747ms
  2> 4710 ERROR (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 4713 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 4722 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]
  2> 4723 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 4723 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 4724 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-06T13:07:22.523108Z
  2> 4736 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001 (source: servlet config: solr.solr.home)
  2> 4760 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001/solr.xml
  2> 9821 WARN  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 10177 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-003/cores
  2> 10179 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 10552 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 10555 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.6.0
  2> 10629 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.ManagedIndexSchemaFactory The schema is configured as managed, but managed schema resource managed-schema.xml  not found - loading non-managed schema schema.xml instead
  2> 10747 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=test
  2> 11183 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
  2> 11184 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
  2> 11195 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
  2> 11228 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.ManagedIndexSchema Upgraded to managed schema at /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001/collection1/conf/managed-schema.xml
  2> 11237 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.ManagedIndexSchemaFactory After upgrading to managed schema, renamed the non-managed schema /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001/collection1/conf/schema.xml to /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001/collection1/conf/schema.xml.bak
  2> 11238 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001/collection1, trusted=true
  2> 11308 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-003/cores/core], dataDir=[/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-003/cores/core/data/]
  2> 11688 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 11973 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, configSetBaseDir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001, hostPort=42931, coreRootDirectory=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-003/cores}
  2> 12278 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.RequestHandlers no default request handler is registered (either '/select' or 'standard')
  2> 12392 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 12460 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 12460 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 12525 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001/tempDir-001/collection1/conf
  2> 12549 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 12571 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> Sul 07, 2023 1:07:20 SUB com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.client.solrj.request.SchemaTest
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@7765f84b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@7765f84b
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@744911dd
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@744911dd
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@3ea5d20b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@3ea5d20b
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:58002 to /127.0.0.1:42873 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@261739b8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@261739b8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@62150f9e
  2> 
  2> "/127.0.0.1:58002 to /127.0.0.1:42873 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5033d673
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5033d673
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "/127.0.0.1:58002 to /127.0.0.1:42873 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6c48e251
  2> 	- locked sun.nio.ch.EPollSelectorImpl@7c6d4059
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@409bf450
  2> 
  2> "SUITE-SchemaTest-seed#[93453D83EA81F2EB]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@63287c95
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@127d251e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@127d251e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:508)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:547)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@2d66becd
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@6979b5a
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.client.solrj.request.SchemaTest.init(SchemaTest.java:123)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@106e7c59
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@30827615
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@288a261e
  2> 
  2> "qtp287454671-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@71120fbc
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5283ed55
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp287454671-18-acceptor-0@6112d098-ServerConnector@10e4fe95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42931}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2b409bcd
  2> 
  2> "qtp287454671-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp287454671-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp287454671-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp287454671-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp287454671-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp287454671-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@506e0e6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-622c87d-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31e09430
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@31e09430
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7cc43e1c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@53f54cba
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@3e9a75af
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@734825d4
  2> 	- locked sun.nio.ch.EPollSelectorImpl@463c6e99
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@7232064
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1bd59c9c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@62bc93c0
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@75feb9de
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@50686cc6
  2> 	- locked sun.nio.ch.EPollSelectorImpl@750588f4
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@135787f4
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@4283cbb4
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@4283cbb4
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@44a762b7
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@44a762b7
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57ba28b9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@57ba28b9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([93453D83EA81F2EB]:0)
  2> NOTE: reproduce with: gradlew test --tests SchemaTest.testCopyFieldAccuracy -Dtests.seed=93453D83EA81F2EB -Dtests.locale=wo-SN -Dtests.timezone=Australia/Victoria -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> Sul 07, 2023 1:07:20 SUB com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> Sul 07, 2023 1:07:21 SUB com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.client.solrj.request.SchemaTest: 
  2>    1) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=25, name=Session-HouseKeeper-622c87d-1, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=20, name=qtp287454671-20, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=22, name=qtp287454671-22, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=19, name=qtp287454671-19, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=15, name=TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB], state=WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:508)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:547)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.client.solrj.request.SchemaTest.init(SchemaTest.java:123)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222040.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=21, name=qtp287454671-21, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   10) Thread[id=18, name=qtp287454671-18-acceptor-0@6112d098-ServerConnector@10e4fe95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42931}, state=RUNNABLE, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=17, name=qtp287454671-17, state=RUNNABLE, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=24, name=qtp287454671-24, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=23, name=qtp287454671-23, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-SchemaTest]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$262/0x000000084033f440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$350/0x0000000840414440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> Sul 07, 2023 1:07:21 SUB com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-SchemaTest]
  2>    2) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-SchemaTest]
  2>    3) Thread[id=25, name=Session-HouseKeeper-622c87d-1, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>    4) Thread[id=20, name=qtp287454671-20, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>    5) Thread[id=22, name=qtp287454671-22, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>    6) Thread[id=19, name=qtp287454671-19, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>    7) Thread[id=15, name=TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB], state=WAITING, group=TGRP-SchemaTest]
  2>    8) Thread[id=21, name=qtp287454671-21, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>    9) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-SchemaTest]
  2>   10) Thread[id=18, name=qtp287454671-18-acceptor-0@6112d098-ServerConnector@10e4fe95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42931}, state=RUNNABLE, group=TGRP-SchemaTest]
  2>   11) Thread[id=17, name=qtp287454671-17, state=RUNNABLE, group=TGRP-SchemaTest]
  2>   12) Thread[id=24, name=qtp287454671-24, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>   13) Thread[id=23, name=qtp287454671-23, state=TIMED_WAITING, group=TGRP-SchemaTest]
  2>   14) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-SchemaTest]
  2>   15) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-SchemaTest]
  2>   16) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-SchemaTest]
  2> 7204170 ERROR (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter.
  2>           => org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:554)
  2> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:554) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4bde8fe3d0f18f03f6ccfc245a842fd0641d6a97 [snapshot build, details omitted]]
  2> 	at org.apache.solr.client.solrj.request.SchemaTest.init(SchemaTest.java:123) [test/:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  2> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  2> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7204669 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.Server Stopped Server@54330f13{STOPPING}[10.0.15,sto=0]
  2> 7204674 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.AbstractConnector Stopped ServerConnector@10e4fe95{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 7204676 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@406992ec{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 7204683 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2084658770
  2> 7204685 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@650ddb64 collection1
  2> 7204686 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@650ddb64
  2> 7204710 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 7204718 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7204727 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7204731 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7204738 INFO  (TEST-SchemaTest.testCopyFieldAccuracy-seed#[93453D83EA81F2EB]) [] o.a.s.SolrTestCaseJ4 ###Ending testCopyFieldAccuracy
  2> 7204741 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 7204742 INFO  (SUITE-SchemaTest-seed#[93453D83EA81F2EB]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.request.SchemaTest_93453D83EA81F2EB-001
  2> Jul 06, 2023 3:07:22 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([93453D83EA81F2EB]:0)
  2> NOTE: test params are: codec=DeflateWithPresetCompressingStoredFieldsData(storedFieldsFormat=Lucene90CompressingStoredFieldsFormat(compressionMode=BEST_COMPRESSION, chunkSize=17943, maxDocsPerChunk=5, blockShift=2), termVectorsFormat=Lucene90CompressingTermVectorsFormat(compressionMode=BEST_COMPRESSION, chunkSize=17943, maxDocsPerChunk=5, blockSize=2)), sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=wo-SN, timezone=Australia/Victoria
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=243998016,total=324534272
  2> NOTE: All tests run in this JVM: [SchemaTest]
  2> NOTE: reproduce with: gradlew test --tests SchemaTest -Dtests.seed=93453D83EA81F2EB -Dtests.locale=wo-SN -Dtests.timezone=Australia/Victoria -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:solrj:test (FAILURE): 1101 test(s), 2 failure(s), 40 skipped

1101 tests completed, 2 failed, 40 skipped

> Task :solr:solrj:test FAILED
> Task :solr:solrj:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.request.SchemaTest.testCopyFieldAccuracy (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.request.SchemaTest.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.request.SchemaTest.testCopyFieldAccuracy" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=93453D83EA81F2EB -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.client.solrj.request.SchemaTest.classMethod (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.request.SchemaTest.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.request.SchemaTest.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=93453D83EA81F2EB -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:solrj:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/solrj/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 28s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m53s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4808 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4808/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22552ms
ok 43 post solr log into solr in 8913ms
ok 44 start solr with ssl in 19919ms
ok 45 start solr with ssl and auth in 22442ms
ok 46 SOLR-11740 check 'solr stop' connection in 28340ms
ok 47 status detects locally running solr in 14666ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2158ms
ok 50 copying files around in 6517ms
ok 51 upconfig in 2624ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m16.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4807 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4807/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22534ms
ok 43 post solr log into solr in 9201ms
ok 44 start solr with ssl in 19713ms
ok 45 start solr with ssl and auth in 23137ms
ok 46 SOLR-11740 check 'solr stop' connection in 28068ms
ok 47 status detects locally running solr in 14656ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2166ms
ok 50 copying files around in 6593ms
ok 51 upconfig in 2619ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m24.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4806 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4806/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22522ms
ok 43 post solr log into solr in 9525ms
ok 44 start solr with ssl in 20380ms
ok 45 start solr with ssl and auth in 22234ms
ok 46 SOLR-11740 check 'solr stop' connection in 28305ms
ok 47 status detects locally running solr in 14367ms
ok 48 status does not expose cli parameters to end user in 206ms
ok 49 listing out files in 2163ms
ok 50 copying files around in 6559ms
ok 51 upconfig in 2640ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m19s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4805 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4805/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22329ms
ok 43 post solr log into solr in 8911ms
ok 44 start solr with ssl in 19926ms
ok 45 start solr with ssl and auth in 22428ms
ok 46 SOLR-11740 check 'solr stop' connection in 28621ms
ok 47 status detects locally running solr in 14393ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2185ms
ok 50 copying files around in 6498ms
ok 51 upconfig in 2598ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 55s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4804 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4804/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23050ms
ok 43 post solr log into solr in 8985ms
ok 44 start solr with ssl in 20075ms
ok 45 start solr with ssl and auth in 21928ms
ok 46 SOLR-11740 check 'solr stop' connection in 28405ms
ok 47 status detects locally running solr in 14426ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2172ms
ok 50 copying files around in 6541ms
ok 51 upconfig in 2638ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 59s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m23.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4803 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4803/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22320ms
ok 43 post solr log into solr in 8922ms
ok 44 start solr with ssl in 19913ms
ok 45 start solr with ssl and auth in 22319ms
ok 46 SOLR-11740 check 'solr stop' connection in 28298ms
ok 47 status detects locally running solr in 14358ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2162ms
ok 50 copying files around in 6516ms
ok 51 upconfig in 2587ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 2s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4802 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4802/

All tests passed

Build Log:
[...truncated 1252 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23176ms
ok 43 post solr log into solr in 9377ms
ok 44 start solr with ssl in 20184ms
ok 45 start solr with ssl and auth in 22532ms
ok 46 SOLR-11740 check 'solr stop' connection in 28435ms
ok 47 status detects locally running solr in 14684ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2184ms
ok 50 copying files around in 6572ms
ok 51 upconfig in 2639ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 4s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4801 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4801/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22815ms
ok 43 post solr log into solr in 9199ms
ok 44 start solr with ssl in 20165ms
ok 45 start solr with ssl and auth in 22287ms
ok 46 SOLR-11740 check 'solr stop' connection in 28184ms
ok 47 status detects locally running solr in 14425ms
ok 48 status does not expose cli parameters to end user in 216ms
ok 49 listing out files in 2180ms
ok 50 copying files around in 6590ms
ok 51 upconfig in 2640ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 12s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m34s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4800 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4800/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22564ms
ok 43 post solr log into solr in 9540ms
ok 44 start solr with ssl in 20205ms
ok 45 start solr with ssl and auth in 23184ms
ok 46 SOLR-11740 check 'solr stop' connection in 28525ms
ok 47 status detects locally running solr in 14413ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2185ms
ok 50 copying files around in 6594ms
ok 51 upconfig in 2626ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 6s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m28s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4799 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4799/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22425ms
ok 43 post solr log into solr in 9139ms
ok 44 start solr with ssl in 19880ms
ok 45 start solr with ssl and auth in 22345ms
ok 46 SOLR-11740 check 'solr stop' connection in 28363ms
ok 47 status detects locally running solr in 14662ms
ok 48 status does not expose cli parameters to end user in 206ms
ok 49 listing out files in 2153ms
ok 50 copying files around in 6472ms
ok 51 upconfig in 2584ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m22s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4798 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4798/

2 tests failed.
FAILED:  org.apache.solr.ltr.TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([223F60F86063AA45]:0)


FAILED:  org.apache.solr.ltr.TestParallelWeightCreation.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([223F60F86063AA45]:0)




Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22834ms
ok 43 post solr log into solr in 9499ms
ok 44 start solr with ssl in 19988ms
ok 45 start solr with ssl and auth in 22196ms
ok 46 SOLR-11740 check 'solr stop' connection in 28439ms
ok 47 status detects locally running solr in 14380ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2166ms
ok 50 copying files around in 6618ms
ok 51 upconfig in 2615ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.TestParallelWeightCreation > testLTRScoringQueryParallelWeightCreationResultOrder FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([223F60F86063AA45]:0)

org.apache.solr.ltr.TestParallelWeightCreation > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([223F60F86063AA45]:0)

org.apache.solr.ltr.TestParallelWeightCreation > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestParallelWeightCreation.txt, copied below:
  1> 10:38:07.746 [TEST-TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder-seed#[223F60F86063AA45]] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> sia 05, 2023 2:38:06 ƔETRƆ com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.ltr.TestParallelWeightCreation
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@10012001
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@10012001
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@40592036
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@40592036
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@5b682ff1
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@5b682ff1
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:58654 to /127.0.0.1:39637 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71e24f83
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@71e24f83
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:58654 to /127.0.0.1:39637 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fc4eb8b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fc4eb8b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:58654 to /127.0.0.1:39637 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@161530d5
  2> 	- locked sun.nio.ch.EPollSelectorImpl@c188241
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestParallelWeightCreation-seed#[223F60F86063AA45]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@6f654a16
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder-seed#[223F60F86063AA45]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@3aa52c4f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@3aa52c4f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@57457cc2
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@68697d67
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder(TestParallelWeightCreation.java:26)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@20abe0a1
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@655b2b58
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1984f85d
  2> 
  2> "qtp183640562-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@45746d52
  2> 	- locked sun.nio.ch.EPollSelectorImpl@79498f4f
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp183640562-18-acceptor-0@30c4739d-ServerConnector@71939e1a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:41345}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@47b1d62b
  2> 
  2> "qtp183640562-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp183640562-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp183640562-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp183640562-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp183640562-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp183640562-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@54655dc6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-450e015f-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@329da4e5
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@329da4e5
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@79b660e
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4ed2e858
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@47c9907
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@30a70729
  2> 	- locked sun.nio.ch.EPollSelectorImpl@6fc22754
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@25b4c053
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@33b597b4
  2> 	- locked sun.nio.ch.EPollSelectorImpl@6afc39b2
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@50bb4bea
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@62eae58b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1c6c8605
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@3b05cae4
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@37baa0b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@37baa0b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@8c1227f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@8c1227f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3db25851
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3db25851
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([223F60F86063AA45]:0)
  2> NOTE: reproduce with: gradlew test --tests TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder -Dtests.seed=223F60F86063AA45 -Dtests.locale=ee -Dtests.timezone=Africa/Tripoli -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> sia 05, 2023 2:38:06 ƔETRƆ com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> sia 05, 2023 2:38:07 ƔETRƆ com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.TestParallelWeightCreation: 
  2>    1) Thread[id=19, name=qtp183640562-19, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=18, name=qtp183640562-18-acceptor-0@30c4739d-ServerConnector@71939e1a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:41345}, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=21, name=qtp183640562-21, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=24, name=qtp183640562-24, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=23, name=qtp183640562-23, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=25, name=Session-HouseKeeper-450e015f-1, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=15, name=TEST-TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder-seed#[223F60F86063AA45], state=WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder(TestParallelWeightCreation.java:26)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   12) Thread[id=20, name=qtp183640562-20, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=22, name=qtp183640562-22, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$343/0x000000084043f040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=17, name=qtp183640562-17, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$252/0x000000084030d440.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> sia 05, 2023 2:38:07 ƔETRƆ com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=19, name=qtp183640562-19, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>    2) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestParallelWeightCreation]
  2>    3) Thread[id=18, name=qtp183640562-18-acceptor-0@30c4739d-ServerConnector@71939e1a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:41345}, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>    4) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>    5) Thread[id=21, name=qtp183640562-21, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>    6) Thread[id=24, name=qtp183640562-24, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>    7) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>    8) Thread[id=23, name=qtp183640562-23, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>    9) Thread[id=25, name=Session-HouseKeeper-450e015f-1, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>   10) Thread[id=15, name=TEST-TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder-seed#[223F60F86063AA45], state=WAITING, group=TGRP-TestParallelWeightCreation]
  2>   11) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestParallelWeightCreation]
  2>   12) Thread[id=20, name=qtp183640562-20, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>   13) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>   14) Thread[id=22, name=qtp183640562-22, state=TIMED_WAITING, group=TGRP-TestParallelWeightCreation]
  2>   15) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  2>   16) Thread[id=17, name=qtp183640562-17, state=RUNNABLE, group=TGRP-TestParallelWeightCreation]
  1> 12:38:07.493 [TEST-TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder-seed#[223F60F86063AA45]] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 8594266022db6611a295a7edf467d6ecbd0d6aa6 [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder(TestParallelWeightCreation.java:26) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/modules/ltr/build/tmp/tests-tmp/solr.ltr.TestParallelWeightCreation_223F60F86063AA45-001
  2> Jul 05, 2023 12:38:08 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([223F60F86063AA45]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=101, maxMBSortInHeap=6.466483220526816, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=ee, timezone=Africa/Tripoli
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=258366400,total=324534272
  2> NOTE: All tests run in this JVM: [TestParallelWeightCreation]
  2> NOTE: reproduce with: gradlew test --tests TestParallelWeightCreation -Dtests.seed=223F60F86063AA45 -Dtests.locale=ee -Dtests.timezone=Africa/Tripoli -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 187 test(s), 2 failure(s)

187 tests completed, 2 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestParallelWeightCreation.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.TestParallelWeightCreation.testLTRScoringQueryParallelWeightCreationResultOrder" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=223F60F86063AA45 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.TestParallelWeightCreation.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestParallelWeightCreation.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.TestParallelWeightCreation.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=223F60F86063AA45 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 44s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h3m6s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4797 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4797/

All tests passed

Build Log:
[...truncated 1253 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22976ms
ok 43 post solr log into solr in 9010ms
ok 44 start solr with ssl in 20033ms
ok 45 start solr with ssl and auth in 23116ms
ok 46 SOLR-11740 check 'solr stop' connection in 28103ms
ok 47 status detects locally running solr in 14680ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6597ms
ok 51 upconfig in 2627ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m17s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4796 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4796/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22238ms
ok 43 post solr log into solr in 8775ms
ok 44 start solr with ssl in 19805ms
ok 45 start solr with ssl and auth in 22266ms
ok 46 SOLR-11740 check 'solr stop' connection in 28305ms
ok 47 status detects locally running solr in 14692ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2153ms
ok 50 copying files around in 6553ms
ok 51 upconfig in 2594ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4795 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4795/

All tests passed

Build Log:
[...truncated 1262 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22310ms
ok 43 post solr log into solr in 9017ms
ok 44 start solr with ssl in 19671ms
ok 45 start solr with ssl and auth in 23285ms
ok 46 SOLR-11740 check 'solr stop' connection in 28367ms
ok 47 status detects locally running solr in 14381ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2151ms
ok 50 copying files around in 6528ms
ok 51 upconfig in 2622ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 59s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m20.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4794 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4794/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22402ms
ok 43 post solr log into solr in 9252ms
ok 44 start solr with ssl in 20061ms
ok 45 start solr with ssl and auth in 22679ms
ok 46 SOLR-11740 check 'solr stop' connection in 28120ms
ok 47 status detects locally running solr in 14372ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2181ms
ok 50 copying files around in 6573ms
ok 51 upconfig in 2622ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m32s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4793 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4793/

All tests passed

Build Log:
[...truncated 1261 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22103ms
ok 43 post solr log into solr in 9254ms
ok 44 start solr with ssl in 19847ms
ok 45 start solr with ssl and auth in 22616ms
ok 46 SOLR-11740 check 'solr stop' connection in 28362ms
ok 47 status detects locally running solr in 14392ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2162ms
ok 50 copying files around in 6529ms
ok 51 upconfig in 2614ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 53s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m15s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4792 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4792/

No tests ran.

Build Log:
[...truncated 118 lines...]
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4791 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4791/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22739ms
ok 43 post solr log into solr in 9092ms
ok 44 start solr with ssl in 19667ms
ok 45 start solr with ssl and auth in 22174ms
ok 46 SOLR-11740 check 'solr stop' connection in 28267ms
ok 47 status detects locally running solr in 14606ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2134ms
ok 50 copying files around in 6418ms
ok 51 upconfig in 2573ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 49s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m9s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4790 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4790/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23123ms
ok 43 post solr log into solr in 9314ms
ok 44 start solr with ssl in 19758ms
ok 45 start solr with ssl and auth in 22616ms
ok 46 SOLR-11740 check 'solr stop' connection in 28384ms
ok 47 status detects locally running solr in 14367ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6533ms
ok 51 upconfig in 2610ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m18s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4789 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4789/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22496ms
ok 43 post solr log into solr in 9181ms
ok 44 start solr with ssl in 20067ms
ok 45 start solr with ssl and auth in 22752ms
ok 46 SOLR-11740 check 'solr stop' connection in 28184ms
ok 47 status detects locally running solr in 14496ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2171ms
ok 50 copying files around in 6667ms
ok 51 upconfig in 2676ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 6s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4788 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4788/

All tests passed

Build Log:
[...truncated 1244 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22405ms
ok 43 post solr log into solr in 9155ms
ok 44 start solr with ssl in 19981ms
ok 45 start solr with ssl and auth in 22656ms
ok 46 SOLR-11740 check 'solr stop' connection in 28240ms
ok 47 status detects locally running solr in 14384ms
ok 48 status does not expose cli parameters to end user in 222ms
ok 49 listing out files in 2187ms
ok 50 copying files around in 6638ms
ok 51 upconfig in 2646ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 4s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m27.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4787 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4787/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22343ms
ok 43 post solr log into solr in 9066ms
ok 44 start solr with ssl in 19795ms
ok 45 start solr with ssl and auth in 22288ms
ok 46 SOLR-11740 check 'solr stop' connection in 28266ms
ok 47 status detects locally running solr in 14342ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6440ms
ok 51 upconfig in 2567ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m24s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4786 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4786/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22340ms
ok 43 post solr log into solr in 9108ms
ok 44 start solr with ssl in 19713ms
ok 45 start solr with ssl and auth in 22051ms
ok 46 SOLR-11740 check 'solr stop' connection in 28124ms
ok 47 status detects locally running solr in 14345ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2141ms
ok 50 copying files around in 6469ms
ok 51 upconfig in 2597ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m21s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4785 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4785/

1 tests failed.
FAILED:  org.apache.solr.search.TestCoordinatorRole.testNRTRestart

Error Message:
java.lang.AssertionError: initial error time threshold exceeded

Stack Trace:
java.lang.AssertionError: initial error time threshold exceeded
	at __randomizedtesting.SeedInfo.seed([68AB6ABEA3C7AE2A:983FDA7501DA7CFC]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.solr.search.TestCoordinatorRole.getHostCoreName(TestCoordinatorRole.java:405)
	at org.apache.solr.search.TestCoordinatorRole.testNRTRestart(TestCoordinatorRole.java:330)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 3745 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22532ms
ok 43 post solr log into solr in 8944ms
ok 44 start solr with ssl in 20249ms
ok 45 start solr with ssl and auth in 22053ms
ok 46 SOLR-11740 check 'solr stop' connection in 28077ms
ok 47 status detects locally running solr in 14342ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2159ms
ok 50 copying files around in 6518ms
ok 51 upconfig in 2604ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.search.TestCoordinatorRole.testNRTRestart (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.search.TestCoordinatorRole.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.search.TestCoordinatorRole.testNRTRestart" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=68AB6ABEA3C7AE2A -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 54s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m16s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4784 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4784/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22249ms
ok 43 post solr log into solr in 9027ms
ok 44 start solr with ssl in 19855ms
ok 45 start solr with ssl and auth in 22811ms
ok 46 SOLR-11740 check 'solr stop' connection in 28003ms
ok 47 status detects locally running solr in 14421ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2142ms
ok 50 copying files around in 6506ms
ok 51 upconfig in 2569ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 47s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m7s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4783 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4783/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22622ms
ok 43 post solr log into solr in 9344ms
ok 44 start solr with ssl in 19820ms
ok 45 start solr with ssl and auth in 22267ms
ok 46 SOLR-11740 check 'solr stop' connection in 28448ms
ok 47 status detects locally running solr in 14379ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2159ms
ok 50 copying files around in 6513ms
ok 51 upconfig in 2616ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m20.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4782 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4782/

No tests ran.

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22319ms
ok 43 post solr log into solr in 9311ms
ok 44 start solr with ssl in 19952ms
ok 45 start solr with ssl and auth in 22433ms
ok 46 SOLR-11740 check 'solr stop' connection in 28665ms
ok 47 status detects locally running solr in 14367ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2146ms
ok 50 copying files around in 6508ms
ok 51 upconfig in 2635ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 50s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m11s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: kex_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(235) [Receiver=3.1.3]

Error: kex_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(235) [Receiver=3.1.3]

Error: kex_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(235) [Receiver=3.1.3]

Error: kex_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(235) [Receiver=3.1.3]

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
ERROR: Step ‘Publish JUnit test result report’ failed: No test report files were found. Configuration error?
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4781 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4781/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22927ms
ok 43 post solr log into solr in 9018ms
ok 44 start solr with ssl in 20357ms
ok 45 start solr with ssl and auth in 23004ms
ok 46 SOLR-11740 check 'solr stop' connection in 28019ms
ok 47 status detects locally running solr in 14362ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2150ms
ok 50 copying files around in 6459ms
ok 51 upconfig in 2600ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 55s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m16s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4780 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4780/

2 tests failed.
FAILED:  org.apache.solr.ltr.feature.TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([E8E5D5EFAC2AF57D]:0)


FAILED:  org.apache.solr.ltr.feature.TestFieldLengthFeature.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([E8E5D5EFAC2AF57D]:0)




Build Log:
[...truncated 1240 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22788ms
ok 43 post solr log into solr in 9549ms
ok 44 start solr with ssl in 20049ms
ok 45 start solr with ssl and auth in 22334ms
ok 46 SOLR-11740 check 'solr stop' connection in 28364ms
ok 47 status detects locally running solr in 14485ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2173ms
ok 50 copying files around in 6597ms
ok 51 upconfig in 2617ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.feature.TestFieldLengthFeature > testIfFieldIsEmptyLengthIsZero FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([E8E5D5EFAC2AF57D]:0)

org.apache.solr.ltr.feature.TestFieldLengthFeature > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([E8E5D5EFAC2AF57D]:0)

org.apache.solr.ltr.feature.TestFieldLengthFeature > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestFieldLengthFeature.txt, copied below:
  1> 06:11:13.572 [TEST-TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero-seed#[E8E5D5EFAC2AF57D]] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> Jul 03, 2023 4:11:12 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.ltr.feature.TestFieldLengthFeature
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@1066a7cd
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@1066a7cd
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@433cec40
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@433cec40
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@560e64b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@560e64b
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:45046 to /127.0.0.1:41355 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5cd128ac
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5cd128ac
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:45046 to /127.0.0.1:41355 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c3fcc3f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c3fcc3f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:45046 to /127.0.0.1:41355 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@28c57f14
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2a9586f9
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestFieldLengthFeature-seed#[E8E5D5EFAC2AF57D]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@7f544f4f
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero-seed#[E8E5D5EFAC2AF57D]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@2140d36a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@2140d36a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@df83aae
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@e46d9da
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.feature.TestFieldLengthFeature.before(TestFieldLengthFeature.java:31)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@8e224a5
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@55912cc1
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@19705197
  2> 
  2> "qtp189710669-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@381e0ceb
  2> 	- locked sun.nio.ch.EPollSelectorImpl@64b5d80d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp189710669-18-acceptor-0@ed0d6a6-ServerConnector@66492ac6{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42255}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7fd2c1e5
  2> 
  2> "qtp189710669-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp189710669-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp189710669-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp189710669-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp189710669-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp189710669-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@58ae86f6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-7737f7fe-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@101fa05c
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@101fa05c
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5aa077e2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@331448ba
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@2f12f050
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1fa1e118
  2> 	- locked sun.nio.ch.EPollSelectorImpl@72d1bff0
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@3b3c5959
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2b97c92f
  2> 	- locked sun.nio.ch.EPollSelectorImpl@3a57057f
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@62198364
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2aee9a0c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@553f9e9e
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@7f6e544f
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@4074fba7
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@4074fba7
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@420a622b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@420a622b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1ceabe9c
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1ceabe9c
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([E8E5D5EFAC2AF57D]:0)
  2> NOTE: reproduce with: gradlew test --tests TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero -Dtests.seed=E8E5D5EFAC2AF57D -Dtests.locale=sw-TZ -Dtests.timezone=America/Dominica -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> Jul 03, 2023 4:11:12 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> Jul 03, 2023 4:11:13 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.feature.TestFieldLengthFeature: 
  2>    1) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=21, name=qtp189710669-21, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=17, name=qtp189710669-17, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=22, name=qtp189710669-22, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=19, name=qtp189710669-19, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=25, name=Session-HouseKeeper-7737f7fe-1, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=15, name=TEST-TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero-seed#[E8E5D5EFAC2AF57D], state=WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.feature.TestFieldLengthFeature.before(TestFieldLengthFeature.java:31)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=20, name=qtp189710669-20, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x000000084043fc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=24, name=qtp189710669-24, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=23, name=qtp189710669-23, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   16) Thread[id=18, name=qtp189710669-18-acceptor-0@ed0d6a6-ServerConnector@66492ac6{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42255}, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> Jul 03, 2023 4:11:13 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>    2) Thread[id=21, name=qtp189710669-21, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>    3) Thread[id=17, name=qtp189710669-17, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>    5) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>    6) Thread[id=22, name=qtp189710669-22, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>    7) Thread[id=19, name=qtp189710669-19, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>    8) Thread[id=25, name=Session-HouseKeeper-7737f7fe-1, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>    9) Thread[id=15, name=TEST-TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero-seed#[E8E5D5EFAC2AF57D], state=WAITING, group=TGRP-TestFieldLengthFeature]
  2>   10) Thread[id=20, name=qtp189710669-20, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>   11) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  2>   12) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestFieldLengthFeature]
  2>   13) Thread[id=24, name=qtp189710669-24, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>   14) Thread[id=23, name=qtp189710669-23, state=TIMED_WAITING, group=TGRP-TestFieldLengthFeature]
  2>   15) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestFieldLengthFeature]
  2>   16) Thread[id=18, name=qtp189710669-18-acceptor-0@ed0d6a6-ServerConnector@66492ac6{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42255}, state=RUNNABLE, group=TGRP-TestFieldLengthFeature]
  1> 08:11:13.189 [TEST-TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero-seed#[E8E5D5EFAC2AF57D]] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.feature.TestFieldLengthFeature.before(TestFieldLengthFeature.java:31) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/modules/ltr/build/tmp/tests-tmp/solr.ltr.feature.TestFieldLengthFeature_E8E5D5EFAC2AF57D-001
  2> Jul 03, 2023 8:11:13 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([E8E5D5EFAC2AF57D]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=513, maxMBSortInHeap=7.788820976540943, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=sw-TZ, timezone=America/Dominica
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=258857416,total=324534272
  2> NOTE: All tests run in this JVM: [TestFieldLengthFeature]
  2> NOTE: reproduce with: gradlew test --tests TestFieldLengthFeature -Dtests.seed=E8E5D5EFAC2AF57D -Dtests.locale=sw-TZ -Dtests.timezone=America/Dominica -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 183 test(s), 2 failure(s)

183 tests completed, 2 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.feature.TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestFieldLengthFeature.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.feature.TestFieldLengthFeature.testIfFieldIsEmptyLengthIsZero" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=E8E5D5EFAC2AF57D -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.ltr.feature.TestFieldLengthFeature.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestFieldLengthFeature.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.feature.TestFieldLengthFeature.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=E8E5D5EFAC2AF57D -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 36s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m58s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4779 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4779/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([A75E4F697CF781DA:4F82E28B8F1F9BFA]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 89.28 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1661 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22568ms
ok 43 post solr log into solr in 9229ms
ok 44 start solr with ssl in 19949ms
ok 45 start solr with ssl and auth in 23221ms
ok 46 SOLR-11740 check 'solr stop' connection in 28468ms
ok 47 status detects locally running solr in 14702ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2184ms
ok 50 copying files around in 6636ms
ok 51 upconfig in 2660ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A75E4F697CF781DA -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 7s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m29s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4778 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4778/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22883ms
ok 43 post solr log into solr in 9100ms
ok 44 start solr with ssl in 20402ms
ok 45 start solr with ssl and auth in 22025ms
ok 46 SOLR-11740 check 'solr stop' connection in 28092ms
ok 47 status detects locally running solr in 14719ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6514ms
ok 51 upconfig in 2616ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m18s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4777 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4777/

2 tests failed.
FAILED:  org.apache.solr.cloud.api.collections.TestCollectionAPI.test

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([8BC782D075418555]:0)


FAILED:  org.apache.solr.cloud.api.collections.TestCollectionAPI.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([8BC782D075418555]:0)




Build Log:
[...truncated 1227 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22232ms
ok 43 post solr log into solr in 9401ms
ok 44 start solr with ssl in 20050ms
ok 45 start solr with ssl and auth in 22102ms
ok 46 SOLR-11740 check 'solr stop' connection in 28688ms
ok 47 status detects locally running solr in 14393ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2209ms
ok 50 copying files around in 6541ms
ok 51 upconfig in 2598ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.cloud.api.collections.TestCollectionAPI > test FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([8BC782D075418555]:0)

org.apache.solr.cloud.api.collections.TestCollectionAPI > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([8BC782D075418555]:0)

org.apache.solr.cloud.api.collections.TestCollectionAPI > test suite's output saved to /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.api.collections.TestCollectionAPI.txt, copied below:
  2> 11646 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 11647 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 11649 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log messages
  2> 11650 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching regex: ignore_exception
  2> 11656 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.TestCollectionAPI_8BC782D075418555-001/data-dir-2-001
  2> 11658 WARN  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=2 numCloses=2
  2> 11660 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
  2> 11668 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 11669 INFO  (SUITE-TestCollectionAPI-seed#[8BC782D075418555]-worker) [] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /
  2> 11698 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-3 after mutting 0 log messages
  2> 11699 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-4 for ERROR logs matching regex: ignore_exception
  2> 11712 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 11744 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 0.0.0.0/0.0.0.0:0
  2> 11745 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting server
  2> 11857 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
  2> 11934 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer start zk server on port: 35173
  2> 12339 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12468 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12538 INFO  (zkConnectionManagerCallback-22-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12540 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12541 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12577 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12603 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12628 INFO  (zkConnectionManagerCallback-24-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12629 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12631 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12687 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
  2> 12739 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
  2> 12753 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
  2> 12776 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
  2> 12794 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
  2> 12811 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
  2> 12821 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
  2> 12841 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
  2> 12909 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
  2> 12922 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
  2> 12936 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkTestServer put /tmp/src/solr/solr/core/build/resources/test/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
  2> 12947 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
  2> 14577 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 14579 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 14586 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 14677 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 14710 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@a70c308{/,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 14718 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.e.j.s.AbstractConnector Started ServerConnector@653b9c4b{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39597}
  2> 14722 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.e.j.s.Server Started Server@40ee2bbb{STARTING}[10.0.15,sto=0] @14741ms
  2> 14754 ERROR (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 14756 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 14766 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0
  2> 14768 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 14769 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 14769 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-02T20:16:09.940858Z
  2> 14774 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.TestCollectionAPI_8BC782D075418555-001/control-001 (source: servlet config: solr.solr.home)
  2> 14777 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14819 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 14843 INFO  (zkConnectionManagerCallback-26-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 14844 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 14844 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14962 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.NodeConfig Loading solr.xml from SolrHome (not found in ZooKeeper)
  2> 14963 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.TestCollectionAPI_8BC782D075418555-001/control-001/solr.xml
  2> 16012 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:35173/solr
  2> 16029 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 16066 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 16102 INFO  (zkConnectionManagerCallback-36-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 16103 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 16103 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 16223 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
  2> 16244 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 16270 INFO  (zkConnectionManagerCallback-38-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 16270 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 16465 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 16559 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=false. Solr will be using Overseer based cluster state updates.
  2> 16600 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39597_
  2> 16608 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.Overseer Overseer (id=72057614091485188-127.0.0.1:39597_-n_0000000000) starting
  2> 16813 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39597_
  2> 16814 INFO  (OverseerStateUpdate-72057614091485188-127.0.0.1:39597_-n_0000000000) [n:127.0.0.1:39597_] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:39597_
  2> 16865 INFO  (OverseerStateUpdate-72057614091485188-127.0.0.1:39597_-n_0000000000) [n:127.0.0.1:39597_] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
  2> 16906 WARN  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 17443 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.TestCollectionAPI_8BC782D075418555-001/control-001/cores
  2> 17634 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 17745 INFO  (TEST-TestCollectionAPI.test-seed#[8BC782D075418555]) [n:127.0.0.1:39597_] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/, solr.data.dir=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.TestCollectionAPI_8BC782D075418555-001/tempDir-001/control/data, hostPort=39597, coreRootDirectory=/tmp/src/solr/solr/core/build/tmp/tests-cwd/../../../../../../../../tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cloud.api.collections.TestCollectionAPI_8BC782D075418555-001/control-001/cores}
  2> Yul 02, 2023 10:16:06 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.cloud.api.collections.TestCollectionAPI
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@4d50ad2
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@4d50ad2
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@58c85041
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@58c85041
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@74674210
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@74674210
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:50910 to /127.0.0.1:41721 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25c7ffa6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@25c7ffa6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@20ad4c12
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@1aa7ecca
  2> 
  2> "/127.0.0.1:50910 to /127.0.0.1:41721 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f97e907
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3f97e907
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@76505305
  2> 
  2> "/127.0.0.1:50910 to /127.0.0.1:41721 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@71ae5b27
  2> 	- locked sun.nio.ch.EPollSelectorImpl@3ba944dc
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@71d44a3
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=25 WAITING on java.util.concurrent.ForkJoinPool@2189a6c0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@2189a6c0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "SUITE-TestCollectionAPI-seed#[8BC782D075418555]" ID=32 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@21d6bc27
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]" ID=33 WAITING on java.util.concurrent.CountDownLatch$Sync@2b2b7aa7
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@2b2b7aa7
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@22a2ac59
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@53dc8375
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:347)
  2> 	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1161)
  2> 	at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1134)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021ac40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@3f8f8068
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2b2c343a
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1cdb8562
  2> 
  2> "ZkTestServer Run Thread" ID=34 WAITING on org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@68e4236a
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@68e4236a
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:915)
  2> 	at app//org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:337)
  2> 	at app//org.apache.solr.cloud.ZkTestServer$1.run(ZkTestServer.java:581)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-1" ID=37 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@22a357a8
  2> 	- locked sun.nio.ch.EPollSelectorImpl@6213360b
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.SelectorThread-0" ID=36 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@121d86ac
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5b2b18f6
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:403)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:368)
  2> 
  2> "NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0" ID=38 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@630d1a2c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1855db28
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2> 
  2> "ConnnectionExpirer" ID=35 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:554)
  2> 
  2> "SessionTracker" ID=39 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:163)
  2> 
  2> "SyncThread:0" ID=40 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66030668
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@66030668
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:170)
  2> 
  2> "ProcessThread(sid:0 cport:35173):" ID=41 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@65ec5ef6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@65ec5ef6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:142)
  2> 
  2> "RequestThrottler" ID=42 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e3be41d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6e3be41d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2> 
  2> "NIOWorkerThread-1" ID=43 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5881ee69
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5881ee69
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "NIOWorkerThread-2" ID=44 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5881ee69
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5881ee69
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-SendThread(127.0.0.1:35173)" ID=45 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@264f8357
  2> 	- locked sun.nio.ch.EPollSelectorImpl@21d37286
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2> 	at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2> 	at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-EventThread" ID=46 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10f971ad
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@10f971ad
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "NIOWorkerThread-3" ID=47 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5881ee69
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5881ee69
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "zkConnectionManagerCallback-22-thread-1" ID=48 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47dc377d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@47dc377d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-SendThread(127.0.0.1:35173)" ID=49 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1dd351ea
  2> 	- locked sun.nio.ch.EPollSelectorImpl@27ae5794
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2> 	at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2> 	at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-EventThread" ID=50 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c813ed5
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7c813ed5
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-24-thread-1" ID=51 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5a4089e7
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5a4089e7
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-53" ID=53 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5d9bcf44
  2> 	- locked sun.nio.ch.EPollSelectorImpl@8dc2194
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-54-acceptor-0@65f9ad3f-ServerConnector@653b9c4b{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39597}" ID=54 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7ddec1cc
  2> 
  2> "qtp1531129118-55" ID=55 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-56" ID=56 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-57" ID=57 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-58" ID=58 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-59" ID=59 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1531129118-60" ID=60 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@230f7021
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-1a5b8088-1" ID=61 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@206d6aeb
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@206d6aeb
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-31-thread-1" ID=65 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@49543ff3
  2> 	- locked sun.nio.ch.EPollSelectorImpl@420f57b4
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@7c312628
  2> 
  2> "httpShardExecutor-31-thread-2" ID=66 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1a59a9ac
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5128b23e
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@6c1aae3f
  2> 
  2> "Connection evictor" ID=67 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=68 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-33-thread-1" ID=69 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@39eba44f
  2> 	- locked sun.nio.ch.EPollSelectorImpl@161f631
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@49951a5a
  2> 
  2> "h2sc-33-thread-2" ID=70 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@56223b78
  2> 	- locked sun.nio.ch.EPollSelectorImpl@420fdac0
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@4235a63
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-SendThread(127.0.0.1:35173)" ID=74 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2622c712
  2> 	- locked sun.nio.ch.EPollSelectorImpl@287c7d99
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:136)
  2> 	at app//org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:332)
  2> 	at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1289)
  2> 
  2> "TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-EventThread" ID=75 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f427a15
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6f427a15
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2> 
  2> "zkConnectionManagerCallback-38-thread-1" ID=76 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53a764e7
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@53a764e7
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "OverseerStateUpdate-72057614091485188-127.0.0.1:39597_-n_0000000000" ID=77 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@630d33d6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@630d33d6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2> 	at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "OverseerCollectionConfigSetProcessor-72057614091485188-127.0.0.1:39597_-n_0000000000" ID=78 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28561a9b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@28561a9b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2> 	at app//org.apache.solr.cloud.OverseerTaskQueue.peekTopN(OverseerTaskQueue.java:267)
  2> 	at app//org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:272)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "CloserThread" ID=80 WAITING on java.lang.Object@54384f56
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@54384f56
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([8BC782D075418555]:0)
  2> NOTE: reproduce with: gradlew test --tests TestCollectionAPI.test -Dtests.seed=8BC782D075418555 -Dtests.locale=ha -Dtests.timezone=Atlantic/Azores -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> Yul 02, 2023 10:16:06 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 36 leaked thread(s).
  2> Yul 02, 2023 10:16:07 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 36 threads leaked from SUITE scope at org.apache.solr.cloud.api.collections.TestCollectionAPI: 
  2>    1) Thread[id=76, name=zkConnectionManagerCallback-38-thread-1, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=33, name=TEST-TestCollectionAPI.test-seed#[8BC782D075418555], state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:347)
  2>         at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1161)
  2>         at app//org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1134)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021ac40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=77, name=OverseerStateUpdate-72057614091485188-127.0.0.1:39597_-n_0000000000, state=TIMED_WAITING, group=Overseer state updater.]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.apache.solr.cloud.ZkDistributedQueue.peekElements(ZkDistributedQueue.java:490)
  2>         at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:338)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=38, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:205)
  2>         at app//org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:181)
  2>    5) Thread[id=46, name=TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-EventThread, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2>    6) Thread[id=47, name=NIOWorkerThread-3, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=50, name=TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-EventThread, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2>    8) Thread[id=42, name=RequestThrottler, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.server.RequestThrottler.run(RequestThrottler.java:147)
  2>    9) Thread[id=75, name=TEST-TestCollectionAPI.test-seed#[8BC782D075418555]-EventThread, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:549)
  2>   10) Thread[id=61, name=Session-HouseKeeper-1a5b8088-1, state=TIMED_WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=40, name=SyncThread:0, state=WAITING, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at 

[...truncated too long message...]

Thread[id=70, name=h2sc-33-thread-2, state=RUNNABLE, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=66, name=httpShardExecutor-31-thread-2, state=RUNNABLE, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=65, name=httpShardExecutor-31-thread-1, state=RUNNABLE, group=TGRP-TestCollectionAPI]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$328/0x0000000840424840.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$329/0x0000000840424040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([8BC782D075418555]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=259, maxMBSortInHeap=7.692459069743444, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=ha, timezone=Atlantic/Azores
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=10,free=246245408,total=327155712
  2> NOTE: All tests run in this JVM: [TestDeprecatedFilters, RestartWhileUpdatingTest, TestCollectionAPI]
  2> NOTE: reproduce with: gradlew test --tests TestCollectionAPI -Dtests.seed=8BC782D075418555 -Dtests.locale=ha -Dtests.timezone=Atlantic/Azores -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4806 test(s), 2 failure(s), 236 skipped

4806 tests completed, 2 failed, 236 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.api.collections.TestCollectionAPI.test (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.api.collections.TestCollectionAPI.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.api.collections.TestCollectionAPI.test" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=8BC782D075418555 -Ptests.file.encoding=ISO-8859-1

  - org.apache.solr.cloud.api.collections.TestCollectionAPI.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.api.collections.TestCollectionAPI.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.api.collections.TestCollectionAPI.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=8BC782D075418555 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 3m 55s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h4m16s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4776 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4776/

All tests passed

Build Log:
[...truncated 1253 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22267ms
ok 43 post solr log into solr in 9665ms
ok 44 start solr with ssl in 20435ms
ok 45 start solr with ssl and auth in 22701ms
ok 46 SOLR-11740 check 'solr stop' connection in 28303ms
ok 47 status detects locally running solr in 14373ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2206ms
ok 50 copying files around in 6557ms
ok 51 upconfig in 2573ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 59s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m20.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4775 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4775/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22970ms
ok 43 post solr log into solr in 8925ms
ok 44 start solr with ssl in 20484ms
ok 45 start solr with ssl and auth in 22995ms
ok 46 SOLR-11740 check 'solr stop' connection in 28143ms
ok 47 status detects locally running solr in 14380ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2171ms
ok 50 copying files around in 6532ms
ok 51 upconfig in 2606ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 49s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m10s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4774 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4774/

1 tests failed.
FAILED:  org.apache.solr.cli.TestSolrCLIRunExample.testInteractiveSolrCloudExample

Error Message:
java.lang.AssertionError: After running Solr cloud example, test collection 'testCloudExamplePrompt' not found in Solr at: http://localhost:39895/solr; tool output: 
Welcome to the SolrCloud example!

This interactive session will help you launch a SolrCloud cluster on your local workstation.
To begin, how many Solr nodes would you like to run in your local cluster? (specify 1-4 nodes) [2]: 
Ok, let's start up 1 Solr nodes for your example SolrCloud cluster.
Please enter the port for node1 [8983]: 
Oops! Looks like port 39895 is already being used by another process. Please choose a different port.
Please enter a port for node 1 [8983]: 


Stack Trace:
java.lang.AssertionError: After running Solr cloud example, test collection 'testCloudExamplePrompt' not found in Solr at: http://localhost:39895/solr; tool output: 
Welcome to the SolrCloud example!

This interactive session will help you launch a SolrCloud cluster on your local workstation.
To begin, how many Solr nodes would you like to run in your local cluster? (specify 1-4 nodes) [2]: 
Ok, let's start up 1 Solr nodes for your example SolrCloud cluster.
Please enter the port for node1 [8983]: 
Oops! Looks like port 39895 is already being used by another process. Please choose a different port.
Please enter a port for node 1 [8983]: 

	at __randomizedtesting.SeedInfo.seed([585C769DE2D99665:832D9657D5AC5303]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.apache.solr.cli.TestSolrCLIRunExample.testInteractiveSolrCloudExample(TestSolrCLIRunExample.java:507)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 1227 lines...]
  2> ERROR: access denied ("java.io.FilePermission" "/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cli.TestSolrCLIRunExample_585C769DE2D99665-001/tempDir-001/failExecuteScript" "execute")
  2> 
  2> 4615 INFO  (TEST-TestSolrCLIRunExample.testFailExecuteScript-seed#[585C769DE2D99665]) [] o.a.s.SolrTestCaseJ4 ###Ending testFailExecuteScript
  2> 4631 INFO  (TEST-TestSolrCLIRunExample.testInteractiveSolrCloudExample-seed#[585C769DE2D99665]) [] o.a.s.SolrTestCaseJ4 ###Starting testInteractiveSolrCloudExample
  2> 
  2> ERROR: access denied ("java.net.SocketPermission" "127.0.0.1:2" "connect,resolve")
  2> 
  2> 5540 INFO  (TEST-TestSolrCLIRunExample.testInteractiveSolrCloudExample-seed#[585C769DE2D99665]) [] o.a.s.SolrTestCaseJ4 ###Ending testInteractiveSolrCloudExample
   >     java.lang.AssertionError: After running Solr cloud example, test collection 'testCloudExamplePrompt' not found in Solr at: http://localhost:39895/solr; tool output: 
   >     Welcome to the SolrCloud example!
   > 
   >     This interactive session will help you launch a SolrCloud cluster on your local workstation.
   >     To begin, how many Solr nodes would you like to run in your local cluster? (specify 1-4 nodes) [2]: 
   >     Ok, let's start up 1 Solr nodes for your example SolrCloud cluster.
   >     Please enter the port for node1 [8983]: 
   >     Oops! Looks like port 39895 is already being used by another process. Please choose a different port.
   >     Please enter a port for node 1 [8983]: 
   >         at __randomizedtesting.SeedInfo.seed([585C769DE2D99665:832D9657D5AC5303]:0)
   >         at org.junit.Assert.fail(Assert.java:89)
   >         at org.apache.solr.cli.TestSolrCLIRunExample.testInteractiveSolrCloudExample(TestSolrCLIRunExample.java:507)
   >         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >         at java.base/java.lang.reflect.Method.invoke(Method.java:566)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
   >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
   >         at org.junit.rules.RunRules.evaluate(RunRules.java:20)
   >         at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
   >         at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
   >         at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
   >         at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
   >         at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
   >         at org.junit.rules.RunRules.evaluate(RunRules.java:20)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
   >         at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
   >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
   >         at org.junit.rules.RunRules.evaluate(RunRules.java:20)
   >         at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
   >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   >         at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
   >         at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
   >         at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
   >         at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
   >         at org.junit.rules.RunRules.evaluate(RunRules.java:20)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
   >         at java.base/java.lang.Thread.run(Thread.java:829)
  2> NOTE: reproduce with: gradlew test --tests TestSolrCLIRunExample.testInteractiveSolrCloudExample -Dtests.seed=585C769DE2D99665 -Dtests.locale=th -Dtests.timezone=Asia/Yangon -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> 5566 INFO  (SUITE-TestSolrCLIRunExample-seed#[585C769DE2D99665]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 5566 INFO  (SUITE-TestSolrCLIRunExample-seed#[585C769DE2D99665]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.cli.TestSolrCLIRunExample_585C769DE2D99665-001
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=339, maxMBSortInHeap=6.620094475487342, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=th, timezone=Asia/Yangon
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=272754440,total=324534272
  2> NOTE: All tests run in this JVM: [TestSolrCLIRunExample]


> Task :solr:packaging:integrationTests
ok 4 create for non cloud mode in 20126ms

> Task :solr:core:test
WARNING: Test org.apache.solr.TestRandomFaceting wrote 12,936,770 bytes of output.

> Task :solr:packaging:integrationTests
ok 5 create for cloud mode in 20765ms
ok 6 create collection in 10325ms
ok 7 reject d option with invalid config dir in 1587ms
ok 8 accept d option with builtin config in 7503ms
ok 9 accept d option with explicit path to config in 6633ms

> Task :solr:modules:hdfs:test
:solr:modules:hdfs:test (SUCCESS): 102 test(s), 54 skipped

> Task :solr:modules:hdfs:wipeTaskTemp
> Task :solr:modules:hdfs:check

> Task :solr:core:test
WARNING: Test org.apache.solr.schema.TestCloudSchemaless wrote 13,247,260 bytes of output.
WARNING: Test org.apache.solr.search.facet.TestJsonFacetRefinement wrote 27,851,684 bytes of output.

> Task :solr:packaging:integrationTests
ok 10 accept n option as config name in 6176ms
ok 11 allow config reuse when n option specifies same config in 10693ms
ok 12 create multisharded collections when s provided in 5857ms
ok 13 create replicated collections when rf provided in 6699ms
ok 14 can delete collections in 9565ms
ok 15 collection delete also deletes zk config in 8001ms

> Task :solr:core:test
:solr:core:test (FAILURE): 4807 test(s), 1 failure(s), 217 skipped

4807 tests completed, 1 failed, 217 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

> Task :solr:packaging:integrationTests
ok 16 deletes accompanying zk config with nondefault name in 8689ms
ok 17 deleteConfig option can opt to leave config in zk in 7961ms
ok 18 SOLR-16755 test works with noprompt in 32664ms
ok 19 Check export command in 29271ms
ok 20 export fails on non cloud mode in 17161ms
ok 21 healthcheck on cloud solr in 21664ms
ok 22 start help flag prints help in 149ms
ok 23 stop help flag prints help in 143ms
ok 24 restart help flag prints help in 147ms
ok 25 status help flag prints help in 47ms # skip Currently the status -help flag doesn't return nice help text!
ok 26 healthcheck help flag prints help in 142ms
ok 27 create help flag prints help in 140ms
ok 28 createcore help flag prints help in 143ms
ok 29 createcollection help flag prints help in 144ms
ok 30 delete help flag prints help in 142ms
ok 31 version help flag prints help in 46ms # skip Currently the version -help flag doesn't return nice help text!
ok 32 zk help flag prints help in 148ms
ok 33 auth help flag prints help in 144ms
ok 34 assert help flag prints help in 47ms # skip Currently the assert -help flag doesn't return nice help text!
ok 35 SQL Module in 23436ms
ok 36 Hadoop-Auth Module: KerberosPlugin Classloading in 26058ms
ok 37 icu collation in analysis-extras module in 19594ms
ok 38 package detects no running solr in 225ms
ok 39 lifecycle of package in 16676ms
ok 40 deploying and undeploying of packages in 20062ms
not ok 41 Affinity placement plugin using sysprop in 18815ms
# (from function `collection_exists' in file test/bats_helper.bash, line 93,
#  in test file test/test_placement_plugin.bats, line 36)
#   `collection_exists COLL_NAME' failed
# WARN  - 2023-07-02 14:38:45.971; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN  - 2023-07-02 14:38:46.028; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Please find the SOLR_HOME snapshot for failed test #1 at: /tmp/src/solr/solr/packaging/build/test-output/failure-snapshots/41-1
# WARN  - 2023-07-02 14:38:47.074; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN  - 2023-07-02 14:38:47.130; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# Last output:
# WARNING: Using _default configset with data driven schema functionality. NOT RECOMMENDED for production use.
#          To turn off: bin/solr config -c COLL_NAME -p 8983 -action set-user-property -property update.autoCreateFields -value false
# WARN  - 2023-07-02 14:38:44.241; org.apache.solr.common.cloud.SolrZkClient; Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
# WARN  - 2023-07-02 14:38:44.294; org.apache.solr.common.cloud.SolrZkClient; Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
#
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22992ms
ok 43 post solr log into solr in 9052ms
ok 44 start solr with ssl in 20068ms
ok 45 start solr with ssl and auth in 22472ms
ok 46 SOLR-11740 check 'solr stop' connection in 28351ms
ok 47 status detects locally running solr in 14423ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2166ms
ok 50 copying files around in 6517ms
ok 51 upconfig in 2645ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.cli.TestSolrCLIRunExample.testInteractiveSolrCloudExample (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cli.TestSolrCLIRunExample.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cli.TestSolrCLIRunExample.testInteractiveSolrCloudExample" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=585C769DE2D99665 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 53s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m13s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4773 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4773/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([D4563F791840AAC2:3C8A929BEBA8B0E2]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 85.6 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1648 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22274ms
ok 43 post solr log into solr in 8842ms
ok 44 start solr with ssl in 20125ms
ok 45 start solr with ssl and auth in 22333ms
ok 46 SOLR-11740 check 'solr stop' connection in 28078ms
ok 47 status detects locally running solr in 14351ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2149ms
ok 50 copying files around in 6488ms
ok 51 upconfig in 2590ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=D4563F791840AAC2 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 49s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m10s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4772 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4772/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22492ms
ok 43 post solr log into solr in 9526ms
ok 44 start solr with ssl in 20076ms
ok 45 start solr with ssl and auth in 22855ms
ok 46 SOLR-11740 check 'solr stop' connection in 28121ms
ok 47 status detects locally running solr in 14518ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2184ms
ok 50 copying files around in 6556ms
ok 51 upconfig in 2671ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 6s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m49.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4771 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4771/

All tests passed

Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22991ms
ok 43 post solr log into solr in 9273ms
ok 44 start solr with ssl in 20307ms
ok 45 start solr with ssl and auth in 22562ms
ok 46 SOLR-11740 check 'solr stop' connection in 28879ms
ok 47 status detects locally running solr in 14790ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2201ms
ok 50 copying files around in 6653ms
ok 51 upconfig in 2662ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 3s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m23s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4770 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4770/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22491ms
ok 43 post solr log into solr in 9450ms
ok 44 start solr with ssl in 19561ms
ok 45 start solr with ssl and auth in 22421ms
ok 46 SOLR-11740 check 'solr stop' connection in 28269ms
ok 47 status detects locally running solr in 14644ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2151ms
ok 50 copying files around in 6570ms
ok 51 upconfig in 2628ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 47s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m8s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4769 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4769/

3 tests failed.
FAILED:  org.apache.solr.TestTolerantSearch.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([B6DD826A53A0BCAB]:0)


FAILED:  org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats

Error Message:
java.lang.AssertionError: Key SEARCHER.searcher.indexVersion not found in registry solr.core.collection1

Stack Trace:
java.lang.AssertionError: Key SEARCHER.searcher.indexVersion not found in registry solr.core.collection1
	at __randomizedtesting.SeedInfo.seed([B6DD826A53A0BCAB:7943E753DC51D4F4]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.apache.solr.handler.admin.StatsReloadRaceTest.requestMetrics(StatsReloadRaceTest.java:157)
	at org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats(StatsReloadRaceTest.java:81)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)


FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([B6DD826A53A0BCAB:5E012F88A048A68B]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 87.96 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 2332 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22295ms
ok 43 post solr log into solr in 9205ms
ok 44 start solr with ssl in 20020ms
ok 45 start solr with ssl and auth in 22752ms
ok 46 SOLR-11740 check 'solr stop' connection in 28620ms
ok 47 status detects locally running solr in 14377ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2150ms
ok 50 copying files around in 6562ms
ok 51 upconfig in 2611ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.TestTolerantSearch > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([B6DD826A53A0BCAB]:0)

org.apache.solr.TestTolerantSearch > test suite's output saved to /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.TestTolerantSearch.txt, copied below:
  2> 3517 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 3529 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 3589 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 3627 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/data-dir-1-001
  2> 3766 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
  2> 3815 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 3935 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 ####initCore
  2> 3936 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 ####initCore end
  2> 3937 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.SolrTestCaseJ4 Writing core.properties file to /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-002/cores/core
  2> 4492 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 4495 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 4503 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 4586 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 4615 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e5bed64{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 4637 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.AbstractConnector Started ServerConnector@1d04618d{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39975}
  2> 4641 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.Server Started Server@559e11d{STARTING}[10.0.15,sto=0] @4662ms
  2> 4655 ERROR (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 4655 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 4663 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0
  2> 4663 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 4664 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 4664 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-02T03:15:57.075735Z
  2> 4666 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-001 (source: servlet config: solr.solr.home)
  2> 4686 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-001/solr.xml
  2> 7228 WARN  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 7557 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-002/cores
  2> 7558 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 7928 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 7930 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.6.0
  2> 8110 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=test
  2> 8781 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.D.s.WordDelimiterFilterFactory Solr loaded a deprecated plugin/analysis class [solr.WordDelimiterFilterFactory]. Please consult documentation how to replace it accordingly.
  2> 8827 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.D.s.CurrencyField Solr loaded a deprecated plugin/analysis class [solr.CurrencyField]. Please consult documentation how to replace it accordingly.
  2> 9334 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 9383 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.0 with uniqueid field id
  2> 9567 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from configset /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-001/collection1, trusted=true
  2> 9726 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-002/cores/core], dataDir=[/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-002/cores/core/data/]
  2> 9813 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=14, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=1.0]
  2> 9802 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, configSetBaseDir=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-001, hostPort=39975, coreRootDirectory=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-002/cores}
  2> 10949 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 11041 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 11041 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 11046 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=30, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=1.0]
  2> 11126 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestTolerantSearch_B6DD826A53A0BCAB-001/tempDir-001/collection1/conf
  2> 11147 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 11181 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> jul. 02, 2023 9:15:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.TestTolerantSearch
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@4b0b7b1c
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@4b0b7b1c
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@205a7b5e
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@205a7b5e
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@1942e8f4
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@1942e8f4
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:54680 to /127.0.0.1:41193 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62bbb99f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@62bbb99f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@5e71477e
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@1aa7ecca
  2> 
  2> "/127.0.0.1:54680 to /127.0.0.1:41193 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1148ae1d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1148ae1d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@76505305
  2> 
  2> "/127.0.0.1:54680 to /127.0.0.1:41193 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1fc3f188
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4bdd0fb9
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@71d44a3
  2> 
  2> "SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@7495301c
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@219c551f
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@219c551f
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@207940d5
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@2036aae3
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100)
  2> 	at app//org.apache.solr.TestTolerantSearch.createThings(TestTolerantSearch.java:64)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@335d99b
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5efd1c31
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@34e890d3
  2> 
  2> "qtp80081956-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@cc8b9b6
  2> 	- locked sun.nio.ch.EPollSelectorImpl@3ae2308a
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp80081956-18-acceptor-0@26db514d-ServerConnector@1d04618d{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39975}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4a7c510d
  2> 
  2> "qtp80081956-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp80081956-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp80081956-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp80081956-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp80081956-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp80081956-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@90bd1f0
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-5a81e769-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4047cc6e
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4047cc6e
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6b0ad28c
  2> 	- locked sun.nio.ch.EPollSelectorImpl@43d602d9
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@444e9d58
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1485cf02
  2> 	- locked sun.nio.ch.EPollSelectorImpl@6a5f5540
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@2b0c4c52
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@577c6435
  2> 	- locked sun.nio.ch.EPollSelectorImpl@24cf64c9
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@454b8ab9
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1e08428d
  2> 	- locked sun.nio.ch.EPollSelectorImpl@eede159
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@37a706ef
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@3a04717d
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@3a04717d
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@65b2a3d8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@65b2a3d8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39f289c9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@39f289c9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
  2> jul. 02, 2023 9:15:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> jul. 02, 2023 9:15:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.TestTolerantSearch: 
  2>    1) Thread[id=23, name=qtp80081956-23, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=15, name=SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker, state=WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100)
  2>         at app//org.apache.solr.TestTolerantSearch.createThings(TestTolerantSearch.java:64)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x0000000840222c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=17, name=qtp80081956-17, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=22, name=qtp80081956-22, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084033f040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840426440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=25, name=Session-HouseKeeper-5a81e769-1, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=21, name=qtp80081956-21, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=20, name=qtp80081956-20, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=24, name=qtp80081956-24, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=19, name=qtp80081956-19, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=18, name=qtp80081956-18-acceptor-0@26db514d-ServerConnector@1d04618d{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39975}, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> jul. 02, 2023 9:15:56 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=23, name=qtp80081956-23, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>    2) Thread[id=15, name=SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker, state=WAITING, group=TGRP-TestTolerantSearch]
  2>    3) Thread[id=17, name=qtp80081956-17, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    4) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    5) Thread[id=22, name=qtp80081956-22, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>    6) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestTolerantSearch]
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    8) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>    9) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2>   10) Thread[id=25, name=Session-HouseKeeper-5a81e769-1, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   11) Thread[id=21, name=qtp80081956-21, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   12) Thread[id=20, name=qtp80081956-20, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   13) Thread[id=24, name=qtp80081956-24, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   14) Thread[id=19, name=qtp80081956-19, state=TIMED_WAITING, group=TGRP-TestTolerantSearch]
  2>   15) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestTolerantSearch]
  2>   16) Thread[id=18, name=qtp80081956-18-acceptor-0@26db514d-ServerConnector@1d04618d{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39975}, state=RUNNABLE, group=TGRP-TestTolerantSearch]
  2> 7204317 ERROR (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter.
  2>           => org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556)
  2> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[main/:?]
  2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [main/:?]
  2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:?]
  2> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:100) [solr-test-framework-9.3.0-SNAPSHOT.jar:?]
  2> 	at org.apache.solr.TestTolerantSearch.createThings(TestTolerantSearch.java:64) [test/:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  2> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  2> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  2> 	at com.carrotsearch.randomi

[...truncated too long message...]

e(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7204818 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.Server Stopped Server@559e11d{STOPPING}[10.0.15,sto=0]
  2> 7204823 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.AbstractConnector Stopped ServerConnector@1d04618d{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 7204826 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6e5bed64{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,STOPPED}
  2> 7204834 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=137590580
  2> 7204836 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@71d00e93 collection1
  2> 7204837 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@71d00e93
  2> 7204862 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 7204868 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7204877 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7204881 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7204887 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 7204888 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> 7204889 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log messages
  2> 7204889 INFO  (SUITE-TestTolerantSearch-seed#[B6DD826A53A0BCAB]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching regex: ignore_exception
  2> Jul 02, 2023 5:15:57 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([B6DD826A53A0BCAB]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=691, maxMBSortInHeap=7.376896127906779, sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=da, timezone=Indian/Mahe
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=215498048,total=324534272
  2> NOTE: All tests run in this JVM: [TestTolerantSearch]
  2> NOTE: reproduce with: gradlew test --tests TestTolerantSearch -Dtests.seed=B6DD826A53A0BCAB -Dtests.locale=da -Dtests.timezone=Indian/Mahe -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4806 test(s), 3 failure(s), 209 skipped

4806 tests completed, 3 failed, 209 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.handler.admin.StatsReloadRaceTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.handler.admin.StatsReloadRaceTest.testParallelReloadAndStats" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B6DD826A53A0BCAB -Ptests.file.encoding=US-ASCII

  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B6DD826A53A0BCAB -Ptests.file.encoding=US-ASCII

  - org.apache.solr.TestTolerantSearch.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.TestTolerantSearch.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.TestTolerantSearch.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B6DD826A53A0BCAB -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 4m 45s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h5m4.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4768 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4768/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23157ms
ok 43 post solr log into solr in 9781ms
ok 44 start solr with ssl in 20191ms
ok 45 start solr with ssl and auth in 23105ms
ok 46 SOLR-11740 check 'solr stop' connection in 28170ms
ok 47 status detects locally running solr in 14752ms
ok 48 status does not expose cli parameters to end user in 216ms
ok 49 listing out files in 2171ms
ok 50 copying files around in 6616ms
ok 51 upconfig in 2670ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 59s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m20s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4767 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4767/

All tests passed

Build Log:
[...truncated 1249 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22508ms
ok 43 post solr log into solr in 9059ms
ok 44 start solr with ssl in 20135ms
ok 45 start solr with ssl and auth in 22595ms
ok 46 SOLR-11740 check 'solr stop' connection in 28483ms
ok 47 status detects locally running solr in 14748ms
ok 48 status does not expose cli parameters to end user in 217ms
ok 49 listing out files in 2199ms
ok 50 copying files around in 6621ms
ok 51 upconfig in 2653ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 55s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4766 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4766/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23086ms
ok 43 post solr log into solr in 8870ms
ok 44 start solr with ssl in 20264ms
ok 45 start solr with ssl and auth in 22718ms
ok 46 SOLR-11740 check 'solr stop' connection in 28117ms
ok 47 status detects locally running solr in 14429ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6569ms
ok 51 upconfig in 2627ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 7s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m33.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4765 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4765/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22251ms
ok 43 post solr log into solr in 9087ms
ok 44 start solr with ssl in 19857ms
ok 45 start solr with ssl and auth in 22561ms
ok 46 SOLR-11740 check 'solr stop' connection in 28076ms
ok 47 status detects locally running solr in 14354ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2168ms
ok 50 copying files around in 6458ms
ok 51 upconfig in 2626ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m20.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4764 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4764/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22251ms
ok 43 post solr log into solr in 8847ms
ok 44 start solr with ssl in 19724ms
ok 45 start solr with ssl and auth in 22359ms
ok 46 SOLR-11740 check 'solr stop' connection in 28014ms
ok 47 status detects locally running solr in 14310ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2156ms
ok 50 copying files around in 6487ms
ok 51 upconfig in 2629ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 53s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m18s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4763 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4763/

2 tests failed.
FAILED:  org.apache.solr.common.cloud.TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([477E83EB1BDBF100]:0)


FAILED:  org.apache.solr.common.cloud.TestDocCollectionWatcher.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([477E83EB1BDBF100]:0)




Build Log:
[...truncated 1241 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22154ms
ok 43 post solr log into solr in 9182ms
ok 44 start solr with ssl in 20467ms
ok 45 start solr with ssl and auth in 22346ms
ok 46 SOLR-11740 check 'solr stop' connection in 28061ms
ok 47 status detects locally running solr in 14657ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2146ms
ok 50 copying files around in 6453ms
ok 51 upconfig in 2563ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:solrj:test

org.apache.solr.common.cloud.TestDocCollectionWatcher > testWaitForStateWatcherIsRetainedOnPredicateFailure FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([477E83EB1BDBF100]:0)

org.apache.solr.common.cloud.TestDocCollectionWatcher > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([477E83EB1BDBF100]:0)

org.apache.solr.common.cloud.TestDocCollectionWatcher > test suite's output saved to /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.common.cloud.TestDocCollectionWatcher.txt, copied below:
  2> 10460 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 10464 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-2 after mutting 0 log messages
  2> 10465 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-3 for ERROR logs matching regex: ignore_exception
  2> 10468 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/data-dir-2-001
  2> 10471 WARN  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
  2> 10473 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
  2> 10476 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 10478 INFO  (SUITE-TestDocCollectionWatcher-seed#[477E83EB1BDBF100]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 10492 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.SolrTestCaseJ4 ###Starting testDeletionsTriggerWatches
  2> 10500 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001
  2> 10505 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.ZkTestServer TEST_PATH() does not exist - solrj test?
  2> 10512 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 10520 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 0.0.0.0/0.0.0.0:0
  2> 10521 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting server
  2> 10582 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
  2> 10633 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.ZkTestServer start zk server on port: 42475
  2> 10961 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11032 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 11622 INFO  (zkConnectionManagerCallback-21-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11622 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11623 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11654 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11669 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 11700 INFO  (zkConnectionManagerCallback-23-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11701 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11701 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11702 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11726 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 11751 INFO  (zkConnectionManagerCallback-25-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 11752 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 11752 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 11913 INFO  (jetty-launcher-26-thread-3) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 11913 INFO  (jetty-launcher-26-thread-2) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 11914 INFO  (jetty-launcher-26-thread-3) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 11915 INFO  (jetty-launcher-26-thread-3) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 11913 INFO  (jetty-launcher-26-thread-1) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 11915 INFO  (jetty-launcher-26-thread-1) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 11915 INFO  (jetty-launcher-26-thread-2) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 11916 INFO  (jetty-launcher-26-thread-2) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 11916 INFO  (jetty-launcher-26-thread-4) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 11916 INFO  (jetty-launcher-26-thread-4) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 11916 INFO  (jetty-launcher-26-thread-4) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 11916 INFO  (jetty-launcher-26-thread-1) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 12025 INFO  (jetty-launcher-26-thread-4) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 12072 INFO  (jetty-launcher-26-thread-4) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@24e6a8e{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 12073 INFO  (jetty-launcher-26-thread-4) [] o.e.j.s.AbstractConnector Started ServerConnector@600cfb4f{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:43055}
  2> 12074 INFO  (jetty-launcher-26-thread-4) [] o.e.j.s.Server Started Server@105a96a1{STARTING}[10.0.15,sto=0] @12097ms
  2> 12079 ERROR (jetty-launcher-26-thread-4) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 12086 INFO  (jetty-launcher-26-thread-4) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 12087 INFO  (jetty-launcher-26-thread-4) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 12087 INFO  (jetty-launcher-26-thread-4) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 12087 INFO  (jetty-launcher-26-thread-4) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 12088 INFO  (jetty-launcher-26-thread-4) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:16.673397Z
  2> 12089 INFO  (jetty-launcher-26-thread-4) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node4 (source: servlet config: solr.solr.home)
  2> 12089 WARN  (jetty-launcher-26-thread-4) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12168 INFO  (jetty-launcher-26-thread-4) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12194 INFO  (zkConnectionManagerCallback-28-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12194 INFO  (jetty-launcher-26-thread-4) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12194 INFO  (jetty-launcher-26-thread-3) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 12195 WARN  (jetty-launcher-26-thread-4) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12260 INFO  (jetty-launcher-26-thread-2) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 12269 INFO  (jetty-launcher-26-thread-2) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@126ee3aa{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 12270 INFO  (jetty-launcher-26-thread-3) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@2cb82892{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 12271 INFO  (jetty-launcher-26-thread-2) [] o.e.j.s.AbstractConnector Started ServerConnector@9e199d8{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:41525}
  2> 12271 INFO  (jetty-launcher-26-thread-3) [] o.e.j.s.AbstractConnector Started ServerConnector@3f622598{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:44673}
  2> 12271 INFO  (jetty-launcher-26-thread-2) [] o.e.j.s.Server Started Server@4c476620{STARTING}[10.0.15,sto=0] @12294ms
  2> 12271 INFO  (jetty-launcher-26-thread-3) [] o.e.j.s.Server Started Server@25f4e2a5{STARTING}[10.0.15,sto=0] @12295ms
  2> 12272 ERROR (jetty-launcher-26-thread-3) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 12272 ERROR (jetty-launcher-26-thread-2) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 12272 INFO  (jetty-launcher-26-thread-2) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 12272 INFO  (jetty-launcher-26-thread-3) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 12272 INFO  (jetty-launcher-26-thread-2) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 12272 INFO  (jetty-launcher-26-thread-3) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 12272 INFO  (jetty-launcher-26-thread-2) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 12272 INFO  (jetty-launcher-26-thread-3) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 12272 INFO  (jetty-launcher-26-thread-2) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 12272 INFO  (jetty-launcher-26-thread-3) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 12272 INFO  (jetty-launcher-26-thread-2) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:16.857723Z
  2> 12272 INFO  (jetty-launcher-26-thread-3) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:16.857759Z
  2> 12273 INFO  (jetty-launcher-26-thread-2) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node2 (source: servlet config: solr.solr.home)
  2> 12273 INFO  (jetty-launcher-26-thread-3) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node3 (source: servlet config: solr.solr.home)
  2> 12273 WARN  (jetty-launcher-26-thread-3) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12273 WARN  (jetty-launcher-26-thread-2) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12275 INFO  (jetty-launcher-26-thread-1) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 12275 INFO  (jetty-launcher-26-thread-4) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 12277 WARN  (jetty-launcher-26-thread-4) [] o.a.s.D.solrxml-zookeeper Loading solr.xml from zookeeper is deprecated. See reference guide for details.
  2> 12281 INFO  (jetty-launcher-26-thread-1) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d0d76c7{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 12286 INFO  (jetty-launcher-26-thread-1) [] o.e.j.s.AbstractConnector Started ServerConnector@2aa29e57{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:46245}
  2> 12286 INFO  (jetty-launcher-26-thread-1) [] o.e.j.s.Server Started Server@7ec0c84{STARTING}[10.0.15,sto=0] @12310ms
  2> 12287 ERROR (jetty-launcher-26-thread-1) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 12287 INFO  (jetty-launcher-26-thread-1) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 12287 INFO  (jetty-launcher-26-thread-1) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 12287 INFO  (jetty-launcher-26-thread-1) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 12287 INFO  (jetty-launcher-26-thread-1) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 12287 INFO  (jetty-launcher-26-thread-1) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:16.872749Z
  2> 12288 INFO  (jetty-launcher-26-thread-1) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node1 (source: servlet config: solr.solr.home)
  2> 12288 INFO  (jetty-launcher-26-thread-2) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12288 WARN  (jetty-launcher-26-thread-1) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12294 INFO  (jetty-launcher-26-thread-4) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 12304 INFO  (jetty-launcher-26-thread-3) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12318 INFO  (jetty-launcher-26-thread-1) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12350 INFO  (zkConnectionManagerCallback-32-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12351 INFO  (jetty-launcher-26-thread-2) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12351 INFO  (zkConnectionManagerCallback-31-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12351 WARN  (jetty-launcher-26-thread-2) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12352 INFO  (jetty-launcher-26-thread-3) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12352 WARN  (jetty-launcher-26-thread-3) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12353 INFO  (zkConnectionManagerCallback-34-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12354 INFO  (jetty-launcher-26-thread-1) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12354 WARN  (jetty-launcher-26-thread-1) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 12357 INFO  (jetty-launcher-26-thread-3) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 12357 INFO  (jetty-launcher-26-thread-2) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 12357 INFO  (jetty-launcher-26-thread-1) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 12362 INFO  (jetty-launcher-26-thread-2) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 12362 INFO  (jetty-launcher-26-thread-3) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 12363 INFO  (jetty-launcher-26-thread-1) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 12595 INFO  (jetty-launcher-26-thread-1) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42475/solr
  2> 12617 INFO  (jetty-launcher-26-thread-1) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 12626 INFO  (jetty-launcher-26-thread-2) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42475/solr
  2> 12627 INFO  (jetty-launcher-26-thread-2) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 12629 INFO  (jetty-launcher-26-thread-4) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42475/solr
  2> 12630 INFO  (jetty-launcher-26-thread-4) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 12645 INFO  (jetty-launcher-26-thread-3) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:42475/solr
  2> 12645 INFO  (jetty-launcher-26-thread-3) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 12668 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12685 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12723 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12724 INFO  (zkConnectionManagerCallback-64-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12740 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12736 INFO  (zkConnectionManagerCallback-74-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12735 INFO  (zkConnectionManagerCallback-68-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12735 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 12749 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12752 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12759 INFO  (zkConnectionManagerCallback-71-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 12760 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 12970 WARN  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 12971 WARN  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 12972 WARN  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 12973 WARN  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 13049 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13049 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13049 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13049 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13072 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:46245_solr
  2> 13078 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44673_solr
  2> 13079 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.Overseer Overseer (id=72057608515354633-127.0.0.1:46245_solr-n_0000000000) starting
  2> 13080 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41525_solr
  2> 13090 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43055_solr
  2> 13151 INFO  (zkCallback-70-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
  2> 13151 INFO  (zkCallback-73-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
  2> 13144 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
  2> 13170 WARN  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 13172 WARN  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 13179 INFO  (zkCallback-63-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
  2> 13182 WARN  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 13322 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.ZkController Publish node=127.0.0.1:46245_solr as DOWN
  2> 13323 INFO  (OverseerStateUpdate-72057608515354633-127.0.0.1:46245_solr-n_0000000000) [n:127.0.0.1:46245_solr] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:46245_solr
  2> 13327 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46245_solr
  2> 13337 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 13337 INFO  (zkCallback-73-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 13337 INFO  (zkCallback-70-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 13338 INFO  (OverseerStateUpdate-72057608515354633-127.0.0.1:46245_solr-n_0000000000) [n:127.0.0.1:46245_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 13361 WARN  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 13398 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 13399 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13657 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 13657 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13673 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 13673 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13708 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node2
  2> 13725 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node3
  2> 13745 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node4
  2> 13769 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 13769 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 13834 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node1
  2> 13900 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 13924 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 14132 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 14136 INFO  (jetty-launcher-26-thread-2) [n:127.0.0.1:41525_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41525, zkHost=127.0.0.1:42475/solr}
  2> 14138 INFO  (jetty-launcher-26-thread-4) [n:127.0.0.1:43055_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43055, zkHost=127.0.0.1:42475/solr}
  2> 14142 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 14168 INFO  (jetty-launcher-26-thread-3) [n:127.0.0.1:44673_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=44673, zkHost=127.0.0.1:42475/solr}
  2> 14187 INFO  (jetty-launcher-26-thread-1) [n:127.0.0.1:46245_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=46245, zkHost=127.0.0.1:42475/solr}
  2> 14209 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=4
  2> 14209 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:41525_solr
  2> 14211 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14224 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 15000ms for client to connect to ZooKeeper
  2> 14245 INFO  (zkConnectionManagerCallback-109-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 14245 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 14246 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14250 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
  2> 14271 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:42475/solr ready
  2> 14273 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:43055_solr
  2> 14273 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:44673_solr
  2> 14273 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:46245_solr
  2> 14274 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14279 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 45000ms for client to connect to ZooKeeper
  2> 14301 INFO  (zkConnectionManagerCallback-111-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 14301 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 14302 WARN  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 14551 INFO  (qtp119957324-100) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Running Collection API locally for CREATE asyncId=null
  2> 14609 INFO  (DistributedCollectionApiCommandExecutor-87-thread-1-processing-127.0.0.1:41525_solr tobedeleted) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.c.a.c.CreateCollectionCmd Create collection tobedeleted
  2> 14862 INFO  (DistributedCollectionApiCommandExecutor-87-thread-1-processing-127.0.0.1:41525_solr tobedeleted) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.c.o.SliceMutator createReplica() {
  2>   "core":"tobedeleted_shard1_replica_n1",
  2>   "node_name":"127.0.0.1:41525_solr",
  2>   "base_url":"http://127.0.0.1:41525/solr",
  2>   "collection":"tobedeleted",
  2>   "shard":"shard1",
  2>   "state":"down",
  2>   "type":"NRT",
  2>   "operation":"ADDREPLICA",
  2>   "waitForFinalState":"false"} 
  2> 15148 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr x:tobedeleted_shard1_replica_n1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=config&newCollection=true&collection=tobedeleted&version=2&replicaType=NRT&coreNodeName=core_node2&name=tobedeleted_shard1_replica_n1&action=CREATE&numShards=1&shard=shard1&wt=javabin
  2> 15243 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.6.0
  2> 15268 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.s.IndexSchema Schema name=test
  2> 15291 WARN  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.s.IndexSchema Field lowerfilt1and2 is not multivalued and destination for multiple copyFields (2)
  2> 15291 WARN  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.s.IndexSchema Field text is not multivalued and destination for multiple copyFields (3)
  2> 15292 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
  2> 15294 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.CoreContainer Creating SolrCore 'tobedeleted_shard1_replica_n1' using configuration from configset config, trusted=true
  2> 15297 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node2/tobedeleted_shard1_replica_n1], dataDir=[/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-001/node2/tobedeleted_shard1_replica_n1/data/]
  2> 15430 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 15445 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
  2> 15445 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
  2> 15453 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 15453 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 15467 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
  2> 15475 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
  2> 15475 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 15481 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1770224479178326016
  2> 15490 INFO  (searcherExecutor-114-thread-1-processing-127.0.0.1:41525_solr tobedeleted_shard1_replica_n1 tobedeleted shard1 core_node2) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> 15520 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tobedeleted/terms/shard1 to Terms{values={core_node2=0}, version=0} for registerTerm
  2> 15528 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/tobedeleted/leaders/shard1
  2> 15576 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
  2> 15576 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
  2> 15577 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:41525/solr/tobedeleted_shard1_replica_n1/
  2> 15587 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
  2> 15589 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.SyncStrategy http://127.0.0.1:41525/solr/tobedeleted_shard1_replica_n1/ has no replicas
  2> 15591 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/tobedeleted/leaders/shard1/leader after winning as /collections/tobedeleted/leader_elect/shard1/election/72057608515354631-core_node2-n_0000000000
  2> 15612 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:41525/solr/tobedeleted_shard1_replica_n1/ shard1
  2> 15634 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr c:tobedeleted s:shard1 r:core_node2 x:tobedeleted_shard1_replica_n1] o.a.s.c.ZkController I am the leader, no recovery necessary
  2> 15661 INFO  (qtp119957324-104) [n:127.0.0.1:41525_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node2&collection.configName=config&newCollection=true&name=tobedeleted_shard1_replica_n1&action=CREATE&numShards=1&collection=tobedeleted&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=514
  2> 15693 INFO  (qtp119957324-100) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
  2> 15704 INFO  (qtp119957324-100) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=config&name=tobedeleted&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=1212
  2> 15764 INFO  (qtp119957324-106) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Running Collection API locally for DELETE asyncId=null
  2> 15784 INFO  (DistributedCollectionApiCommandExecutor-87-thread-2-processing-127.0.0.1:41525_solr tobedeleted) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.c.a.c.CollectionHandlingUtils Executing Collection Cmd=action=UNLOAD&deleteInstanceDir=true&deleteDataDir=true, asyncId=null
  2> 15791 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr x:tobedeleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tobedeleted.shard1.replica_n1 tag=null
  2> 15795 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr x:tobedeleted_shard1_replica_n1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@5f089c15 tobedeleted_shard1_replica_n1
  2> 15796 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr x:tobedeleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.tobedeleted.shard1.replica_n1 tag=SolrCore@5f089c15
  2> 15796 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr x:tobedeleted_shard1_replica_n1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.tobedeleted.shard1.leader tag=SolrCore@5f089c15
  2> 15798 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr x:tobedeleted_shard1_replica_n1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 15856 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr] o.a.s.c.ZkShardTerms Successful update of terms at /collections/tobedeleted/terms/shard1 to Terms{values={}, version=1} for removeTerm
  2> 15879 INFO  (qtp119957324-89) [n:127.0.0.1:41525_solr] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={deleteInstanceDir=true&core=tobedeleted_shard1_replica_n1&qt=/admin/cores&deleteDataDir=true&action=UNLOAD&wt=javabin&version=2} status=0 QTime=89
  2> 15879 INFO  (zkCallback-108-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tobedeleted/state.json] for collection [tobedeleted] has occurred - updating... (live nodes size: [4])
  2> 15882 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tobedeleted/state.json] for collection [tobedeleted] has occurred - updating... (live nodes size: [4])
  2> 15887 INFO  (DistributedCollectionApiCommandExecutor-87-thread-2-processing-127.0.0.1:41525_solr tobedeleted) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.c.c.ZkStateReader /collections/tobedeleted/state.json is deleted, stop watching children
  2> 15894 INFO  (zkCallback-108-thread-1) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/tobedeleted/state.json] for collection [tobedeleted] has occurred - updating... (live nodes size: [4])
  2> 15894 INFO  (zkCallback-108-thread-2) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/tobedeleted/state.json] for collection [tobedeleted] has occurred - updating... (live nodes size: [4])
  2> 15943 INFO  (qtp119957324-106) [n:127.0.0.1:41525_solr c:tobedeleted] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={name=tobedeleted&action=DELETE&wt=javabin&version=2} status=0 QTime=179
  2> 16083 INFO  (jetty-closer-119-thread-1) [] o.e.j.s.Server Stopped Server@4c476620{STOPPING}[10.0.15,sto=0]
  2> 16084 INFO  (jetty-closer-119-thread-3) [] o.e.j.s.Server Stopped Server@25f4e2a5{STOPPING}[10.0.15,sto=0]
  2> 16083 INFO  (jetty-closer-119-thread-2) [] o.e.j.s.Server Stopped Server@105a96a1{STOPPING}[10.0.15,sto=0]
  2> 16087 INFO  (jetty-closer-119-thread-1) [] o.e.j.s.AbstractConnector Stopped ServerConnector@9e199d8{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 16088 INFO  (jetty-closer-119-thread-1) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@126ee3aa{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 16097 INFO  (jetty-closer-119-thread-3) [] o.e.j.s.AbstractConnector Stopped ServerConnector@3f622598{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 16106 INFO  (jetty-closer-119-thread-1) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1434641728
  2> 16097 INFO  (jetty-closer-119-thread-2) [] o.e.j.s.AbstractConnector Stopped ServerConnector@600cfb4f{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 16107 INFO  (jetty-closer-119-thread-4) [] o.e.j.s.Server Stopped Server@7ec0c84{STOPPING}[10.0.15,sto=0]
  2> 16107 INFO  (jetty-closer-119-thread-1) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41525_solr
  2> 16107 INFO  (jetty-closer-119-thread-3) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2cb82892{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 16107 INFO  (jetty-closer-119-thread-2) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@24e6a8e{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 16110 INFO  (jetty-closer-119-thread-1) [] o.a.s.c.ZkController Publish this node as DOWN...
  2> 16111 INFO  (jetty-closer-119-thread-1) [] o.a.s.c.ZkController Publish node=127.0.0.1:41525_solr as DOWN
  2> 16112 INFO  (zkCallback-70-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
  2> 16112 INFO  (zkCallback-73-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
  2> 16112 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
  2> 16112 INFO  (zkCallback-63-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
  2> 16115 INFO  (jetty-closer-119-thread-4) [] o.e.j.s.AbstractConnector Stopped ServerConnector@2aa29e57{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 16115 INFO  (jetty-closer-119-thread-4) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6d0d76c7{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 16145 INFO  (jetty-closer-119-thread-3) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=83664411
  2> 16145 INFO  (jetty-closer-119-thread-3) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:44673_solr
  2> 16147 INFO  (jetty-closer-119-thread-3) [] o.a.s.c.ZkController Publish this node as DOWN...
  2> 16147 INFO  (jetty-closer-119-thread-2) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1005787059
  2> 16147 INFO  (jetty-closer-119-thread-3) [] o.a.s.c.ZkController Publish node=127.0.0.1:44673_solr as DOWN
  2> 16147 INFO  (jetty-closer-119-thread-2) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:43055_solr
  2> 16148 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
  2> 16148 INFO  (zkCallback-63-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
  2> 16148 INFO  (zkCallback-70-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
  2> 16148 INFO  (zkCallback-73-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (2)
  2> 16149 INFO  (jetty-closer-119-thread-4) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1316788295
  2> 16149 INFO  (jetty-closer-119-thread-4) [] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:46245_solr
  2> 16149 INFO  (jetty-closer-119-thread-2) [] o.a.s.c.ZkController Publish this node as DOWN...
  2> 16149 INFO  (jetty-closer-119-thread-2) [] o.a.s.c.ZkController Publish node=127.0.0.1:43055_solr as DOWN
  2> 16150 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
  2> 16150 INFO  (zkCallback-63-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
  2> 16151 INFO  (jetty-closer-119-thread-4) [] o.a.s.c.ZkController Publish this node as DOWN...
  2> 16151 INFO  (jetty-closer-119-thread-4) [] o.a.s.c.ZkController Publish node=127.0.0.1:46245_solr as DOWN
  2> 16151 INFO  (zkCallback-70-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
  2> 16151 INFO  (zkCallback-73-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (0)
  2> 16151 INFO  (zkCallback-63-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 16152 INFO  (zkCallback-67-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 16196 INFO  (jetty-closer-119-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 16197 INFO  (jetty-closer-119-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 16197 INFO  (jetty-closer-119-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 16197 INFO  (jetty-closer-119-thread-1) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 16236 INFO  (jetty-closer-119-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 16237 INFO  (jetty-closer-119-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 16237 INFO  (jetty-closer-119-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 16237 INFO  (jetty-closer-119-thread-4) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 16241 INFO  (jetty-closer-119-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 16241 INFO  (jetty-closer-119-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 16242 INFO  (jetty-closer-119-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 16242 INFO  (jetty-closer-119-thread-2) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 16265 INFO  (jetty-closer-119-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 16265 INFO  (jetty-closer-119-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 16265 INFO  (jetty-closer-119-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 16265 INFO  (jetty-closer-119-thread-3) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 16363 INFO  (closeThreadPool-130-thread-2) [] o.a.s.c.Overseer Overseer (id=72057608515354633-127.0.0.1:46245_solr-n_0000000000) closing
  2> 16366 INFO  (OverseerStateUpdate-72057608515354633-127.0.0.1:46245_solr-n_0000000000) [n:127.0.0.1:46245_solr] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:46245_solr
  2> 16379 INFO  (closeThreadPool-130-thread-1) [] o.a.s.c.Overseer Overseer (id=72057608515354633-127.0.0.1:46245_solr-n_0000000000) closing
  2> 16487 INFO  (jetty-closer-119-thread-4) [] o.a.s.c.Overseer Overseer (id=72057608515354633-127.0.0.1:46245_solr-n_0000000000) closing
  2> 16490 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
  2> 16715 WARN  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Watch limit violations: 
  2> Maximum concurrent create/delete watches above limit:
  2> 
  2> 	5	/solr/aliases.json
  2> 	5	/solr/clusterprops.json
  2> 	4	/solr/packages.json
  2> 	4	/solr/security.json
  2> 	3	/solr/collections/tobedeleted/terms/shard1
  2> 	3	/solr/collections/tobedeleted/state.json
  2> 
  2> Maximum concurrent data watches above limit:
  2> 
  2> 	10	/solr/collections/tobedeleted/state.json
  2> 	3	/solr/overseer_elect/election/72057608515354633-127.0.0.1:46245_solr-n_0000000000
  2> 
  2> Maximum concurrent children watches above limit:
  2> 
  2> 	27	/solr/live_nodes
  2> 	15	/solr/collections
  2> 	8	/solr/collections/tobedeleted/state.json
  2> 
  2> 16740 INFO  (TEST-TestDocCollectionWatcher.testDeletionsTriggerWatches-seed#[477E83EB1BDBF100]) [] o.a.s.SolrTestCaseJ4 ###Ending testDeletionsTriggerWatches
  2> 16758 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.SolrTestCaseJ4 ###Starting testPredicateFailureTimesOut
  2> 16759 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002
  2> 16760 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
  2> 16778 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer client port: 0.0.0.0/0.0.0.0:0
  2> 16779 INFO  (ZkTestServer Run Thread) [] o.a.s.c.ZkTestServer Starting server
  2> 16786 WARN  (ZkTestServer Run Thread) [] o.a.z.s.ServerCnxnFactory maxCnxns is not configured, using default value 0.
  2> 16878 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.ZkTestServer start zk server on port: 39523
  2> 16901 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 16903 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 16941 INFO  (zkConnectionManagerCallback-133-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 16942 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 16942 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 16958 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 16969 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 16996 INFO  (zkConnectionManagerCallback-135-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 16997 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 16997 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 16998 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17009 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17021 INFO  (zkConnectionManagerCallback-137-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17021 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17021 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17137 INFO  (jetty-launcher-138-thread-1) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 17137 INFO  (jetty-launcher-138-thread-1) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 17137 INFO  (jetty-launcher-138-thread-1) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 17143 INFO  (jetty-launcher-138-thread-2) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 17143 INFO  (jetty-launcher-138-thread-2) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 17143 INFO  (jetty-launcher-138-thread-2) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 17140 INFO  (jetty-launcher-138-thread-3) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 17144 INFO  (jetty-launcher-138-thread-3) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 17144 INFO  (jetty-launcher-138-thread-3) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 17150 INFO  (jetty-launcher-138-thread-4) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 17150 INFO  (jetty-launcher-138-thread-4) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 17150 INFO  (jetty-launcher-138-thread-4) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 17277 INFO  (jetty-launcher-138-thread-4) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 17281 INFO  (jetty-launcher-138-thread-4) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6ae0d2be{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 17282 INFO  (jetty-launcher-138-thread-4) [] o.e.j.s.AbstractConnector Started ServerConnector@3b6bad3d{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:38961}
  2> 17283 INFO  (jetty-launcher-138-thread-4) [] o.e.j.s.Server Started Server@1a795849{STARTING}[10.0.15,sto=0] @17306ms
  2> 17283 ERROR (jetty-launcher-138-thread-4) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 17283 INFO  (jetty-launcher-138-thread-4) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 17283 INFO  (jetty-launcher-138-thread-4) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 17283 INFO  (jetty-launcher-138-thread-4) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 17283 INFO  (jetty-launcher-138-thread-4) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 17283 INFO  (jetty-launcher-138-thread-4) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:21.868989Z
  2> 17284 INFO  (jetty-launcher-138-thread-4) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node4 (source: servlet config: solr.solr.home)
  2> 17285 WARN  (jetty-launcher-138-thread-4) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17335 INFO  (jetty-launcher-138-thread-4) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17353 INFO  (jetty-launcher-138-thread-3) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 17405 INFO  (jetty-launcher-138-thread-3) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@17eecb45{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 17407 INFO  (jetty-launcher-138-thread-3) [] o.e.j.s.AbstractConnector Started ServerConnector@34dca0a4{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39715}
  2> 17407 INFO  (jetty-launcher-138-thread-3) [] o.e.j.s.Server Started Server@15ce55d0{STARTING}[10.0.15,sto=0] @17431ms
  2> 17408 ERROR (jetty-launcher-138-thread-3) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 17408 INFO  (jetty-launcher-138-thread-3) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 17408 INFO  (jetty-launcher-138-thread-3) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 17408 INFO  (jetty-launcher-138-thread-3) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 17408 INFO  (jetty-launcher-138-thread-3) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 17408 INFO  (jetty-launcher-138-thread-3) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:21.993452Z
  2> 17408 INFO  (jetty-launcher-138-thread-3) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node3 (source: servlet config: solr.solr.home)
  2> 17409 WARN  (jetty-launcher-138-thread-3) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17413 INFO  (zkConnectionManagerCallback-140-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17414 INFO  (jetty-launcher-138-thread-4) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17414 WARN  (jetty-launcher-138-thread-4) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17435 INFO  (jetty-launcher-138-thread-3) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17470 INFO  (jetty-launcher-138-thread-4) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 17475 INFO  (jetty-launcher-138-thread-4) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 17480 INFO  (jetty-launcher-138-thread-1) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 17495 INFO  (jetty-launcher-138-thread-1) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@452a01ea{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 17496 INFO  (jetty-launcher-138-thread-1) [] o.e.j.s.AbstractConnector Started ServerConnector@763a0745{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:41297}
  2> 17496 INFO  (jetty-launcher-138-thread-1) [] o.e.j.s.Server Started Server@6fada63{STARTING}[10.0.15,sto=0] @17520ms
  2> 17497 ERROR (jetty-launcher-138-thread-1) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 17497 INFO  (jetty-launcher-138-thread-1) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 17497 INFO  (jetty-launcher-138-thread-1) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 17497 INFO  (jetty-launcher-138-thread-1) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 17497 INFO  (jetty-launcher-138-thread-1) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 17497 INFO  (jetty-launcher-138-thread-1) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:22.082400Z
  2> 17497 INFO  (jetty-launcher-138-thread-1) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node1 (source: servlet config: solr.solr.home)
  2> 17498 WARN  (jetty-launcher-138-thread-1) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17499 INFO  (jetty-launcher-138-thread-2) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 17507 INFO  (jetty-launcher-138-thread-2) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@194abb87{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 17508 INFO  (jetty-launcher-138-thread-2) [] o.e.j.s.AbstractConnector Started ServerConnector@3cfac71{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:42707}
  2> 17508 INFO  (jetty-launcher-138-thread-2) [] o.e.j.s.Server Started Server@22dd5905{STARTING}[10.0.15,sto=0] @17532ms
  2> 17508 ERROR (jetty-launcher-138-thread-2) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 17508 INFO  (jetty-launcher-138-thread-2) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 17508 INFO  (jetty-launcher-138-thread-2) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 17509 INFO  (jetty-launcher-138-thread-2) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in cloud mode on port null
  2> 17509 INFO  (jetty-launcher-138-thread-2) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 17509 INFO  (jetty-launcher-138-thread-2) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-07-01T13:20:22.094084Z
  2> 17509 INFO  (jetty-launcher-138-thread-2) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node2 (source: servlet config: solr.solr.home)
  2> 17509 WARN  (jetty-launcher-138-thread-2) [] o.a.s.c.c.SolrZkClient Using default ZkCredentialsInjector. ZkCredentialsInjector is not secure, it creates an empty list of credentials which leads to 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17540 INFO  (jetty-launcher-138-thread-1) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17540 INFO  (zkConnectionManagerCallback-142-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17541 INFO  (jetty-launcher-138-thread-3) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17541 WARN  (jetty-launcher-138-thread-3) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17559 INFO  (zkConnectionManagerCallback-144-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17560 INFO  (jetty-launcher-138-thread-1) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17560 WARN  (jetty-launcher-138-thread-1) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17591 INFO  (jetty-launcher-138-thread-2) [] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17595 INFO  (jetty-launcher-138-thread-3) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 17598 INFO  (jetty-launcher-138-thread-1) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 17600 INFO  (jetty-launcher-138-thread-3) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 17602 INFO  (jetty-launcher-138-thread-1) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 17612 INFO  (zkConnectionManagerCallback-146-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17612 INFO  (jetty-launcher-138-thread-2) [] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17612 WARN  (jetty-launcher-138-thread-2) [] o.a.s.c.c.SolrZkClient Using default ZkACLProvider. DefaultZkACLProvider is not secure, it creates 'OPEN_ACL_UNSAFE' ACLs to Zookeeper nodes
  2> 17616 INFO  (jetty-launcher-138-thread-2) [] o.a.s.c.NodeConfig solr.xml found in ZooKeeper. Loading...
  2> 17621 INFO  (jetty-launcher-138-thread-2) [] o.a.s.c.SolrXmlConfig Metrics collection is disabled.
  2> 17677 INFO  (jetty-launcher-138-thread-4) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39523/solr
  2> 17677 INFO  (jetty-launcher-138-thread-4) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 17723 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17793 INFO  (zkConnectionManagerCallback-156-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17793 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17843 INFO  (jetty-launcher-138-thread-3) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39523/solr
  2> 17843 INFO  (jetty-launcher-138-thread-3) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 17904 WARN  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 17904 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17915 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 17937 INFO  (jetty-launcher-138-thread-1) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39523/solr
  2> 17938 INFO  (jetty-launcher-138-thread-1) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 17939 INFO  (zkConnectionManagerCallback-180-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 17940 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 17947 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38961_solr
  2> 17948 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 17957 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.Overseer Overseer (id=72057608917680135-127.0.0.1:38961_solr-n_0000000000) starting
  2> 17997 INFO  (jetty-launcher-138-thread-2) [] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39523/solr
  2> 17997 INFO  (jetty-launcher-138-thread-2) [] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18005 WARN  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 18023 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18031 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39715_solr
  2> 18034 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.ZkController Publish node=127.0.0.1:38961_solr as DOWN
  2> 18035 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38961_solr
  2> 18038 WARN  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 18046 WARN  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 18056 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.c.ConnectionManager Waiting up to 30000ms for client to connect to ZooKeeper
  2> 18097 INFO  (zkConnectionManagerCallback-184-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 18098 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 18102 INFO  (OverseerStateUpdate-72057608917680135-127.0.0.1:38961_solr-n_0000000000) [n:127.0.0.1:38961_solr] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:38961_solr
  2> 18103 INFO  (zkCallback-179-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
  2> 18103 INFO  (zkCallback-155-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
  2> 18121 INFO  (zkConnectionManagerCallback-190-thread-1) [] o.a.s.c.c.ConnectionManager zkClient has connected
  2> 18122 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
  2> 18175 WARN  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 18178 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
  2> 18179 WARN  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.ZkController Contents of zookeeper /security.json are world-readable; consider setting up ACLs as described in https://solr.apache.org/guide/solr/latest/deployment-guide/zookeeper-access-control.html
  2> 18181 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
  2> 18192 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18196 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18201 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.ZkController Publish node=127.0.0.1:41297_solr as DOWN
  2> 18202 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41297_solr
  2> 18215 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 18215 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18217 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.ZkController Publish node=127.0.0.1:42707_solr as DOWN
  2> 18217 INFO  (zkCallback-179-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
  2> 18217 INFO  (zkCallback-155-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
  2> 18219 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42707_solr
  2> 18220 INFO  (zkCallback-183-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
  2> 18220 INFO  (zkCallback-189-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
  2> 18223 INFO  (zkCallback-179-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 18224 INFO  (zkCallback-155-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 18225 INFO  (zkCallback-189-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 18226 WARN  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 18227 INFO  (zkCallback-183-thread-1) [] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
  2> 18232 WARN  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 18233 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 18233 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18233 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 18233 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18319 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node2
  2> 18320 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.a.c.DistributedCollectionConfigSetCommandRunner Creating DistributedCollectionConfigSetCommandRunner. Collection and ConfigSet APIs are running distributed (not Overseer based)
  2> 18320 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.DistributedClusterStateUpdater Creating DistributedClusterStateUpdater with useDistributedStateUpdate=true. Solr will be using distributed cluster state updates.
  2> 18327 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node3
  2> 18327 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node1
  2> 18413 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.common.cloud.TestDocCollectionWatcher_477E83EB1BDBF100-001/tempDir-002/node4
  2> 18439 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 18579 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 18595 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 18605 INFO  (jetty-launcher-138-thread-3) [n:127.0.0.1:39715_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39715, zkHost=127.0.0.1:39523/solr}
  2> 18627 INFO  (jetty-launcher-138-thread-1) [n:127.0.0.1:41297_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41297, zkHost=127.0.0.1:39523/solr}
  2> 18639 INFO  (jetty-launcher-138-thread-2) [n:127.0.0.1:42707_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=42707, zkHost=127.0.0.1:39523/solr}
  2> 18655 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 18685 INFO  (jetty-launcher-138-thread-4) [n:127.0.0.1:38961_solr] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38961, zkHost=127.0.0.1:39523/solr}
  2> 18696 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=4
  2> 18697 INFO  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF100]) [] o.a.s.c.MiniSolrCloudCluster waitForNode: 127.0.0.1:39715_solr
  2> 18697 WARN  (TEST-TestDocCollectionWatcher.testPredicateFailureTimesOut-seed#[477E83EB1BDBF

[...truncated too long message...]

tiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$368/0x0000000840445840.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$369/0x0000000840445040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   70) Thread[id=506, name=zkConnectionManagerCallback-281-thread-1, state=WAITING, group=TGRP-TestDocCollectionWatcher]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   71) Thread[id=521, name=CloserThread, state=WAITING, group=TGRP-TestDocCollectionWatcher]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   72) Thread[id=431, name=qtp1410641918-431, state=TIMED_WAITING, group=TGRP-TestDocCollectionWatcher]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   73) Thread[id=416, name=TEST-TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure-seed#[477E83EB1BDBF100]-SendThread(127.0.0.1:44137), state=TIMED_WAITING, group=TGRP-TestDocCollectionWatcher]
  2>         at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>         at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:362)
  2>         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
  2>   74) Thread[id=478, name=httpShardExecutor-271-thread-2, state=RUNNABLE, group=TGRP-TestDocCollectionWatcher]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$368/0x0000000840445840.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$369/0x0000000840445040.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   75) Thread[id=500, name=jetty-launcher-244-thread-1-SendThread(127.0.0.1:44137), state=TIMED_WAITING, group=TGRP-TestDocCollectionWatcher]
  2>         at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2>         at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:362)
  2>         at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1204)
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([477E83EB1BDBF100]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=1434, maxMBSortInHeap=5.971520275995656, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=en-HK, timezone=Asia/Brunei
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=87,free=195414696,total=329777152
  2> NOTE: All tests run in this JVM: [JettyWebappTest, TestDocCollectionWatcher]
  2> NOTE: reproduce with: gradlew test --tests TestDocCollectionWatcher -Dtests.seed=477E83EB1BDBF100 -Dtests.locale=en-HK -Dtests.timezone=Asia/Brunei -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:solrj:test (FAILURE): 1131 test(s), 2 failure(s), 40 skipped

1131 tests completed, 2 failed, 40 skipped

> Task :solr:solrj:test FAILED
> Task :solr:solrj:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.common.cloud.TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.common.cloud.TestDocCollectionWatcher.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.common.cloud.TestDocCollectionWatcher.testWaitForStateWatcherIsRetainedOnPredicateFailure" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=477E83EB1BDBF100 -Ptests.file.encoding=US-ASCII

  - org.apache.solr.common.cloud.TestDocCollectionWatcher.classMethod (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.common.cloud.TestDocCollectionWatcher.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.common.cloud.TestDocCollectionWatcher.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=477E83EB1BDBF100 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:solrj:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/solrj/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 42s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h3m8.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4762 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4762/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22405ms
ok 43 post solr log into solr in 8947ms
ok 44 start solr with ssl in 20005ms
ok 45 start solr with ssl and auth in 22834ms
ok 46 SOLR-11740 check 'solr stop' connection in 28159ms
ok 47 status detects locally running solr in 14423ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2168ms
ok 50 copying files around in 6544ms
ok 51 upconfig in 2669ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4761 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4761/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22467ms
ok 43 post solr log into solr in 8661ms
ok 44 start solr with ssl in 19776ms
ok 45 start solr with ssl and auth in 22978ms
ok 46 SOLR-11740 check 'solr stop' connection in 28034ms
ok 47 status detects locally running solr in 14340ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2159ms
ok 50 copying files around in 6528ms
ok 51 upconfig in 2615ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 53s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m19s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4760 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4760/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22443ms
ok 43 post solr log into solr in 8835ms
ok 44 start solr with ssl in 20193ms
ok 45 start solr with ssl and auth in 22169ms
ok 46 SOLR-11740 check 'solr stop' connection in 28093ms
ok 47 status detects locally running solr in 14339ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6526ms
ok 51 upconfig in 2619ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 52s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m19.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4759 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4759/

1 tests failed.
FAILED:  org.apache.solr.cloud.ReindexCollectionTest.testAbort

Error Message:
java.lang.AssertionError: {responseHeader={status=0, QTime=28}, status={state=running, msg=found [76b8b203-0f1b-49bf-b013-71a70bc1e7aa] in running tasks}}

Stack Trace:
java.lang.AssertionError: {responseHeader={status=0, QTime=28}, status={state=running, msg=found [76b8b203-0f1b-49bf-b013-71a70bc1e7aa] in running tasks}}
	at __randomizedtesting.SeedInfo.seed([A989B10F32D4C3D9:CFEA0698460E51A0]:0)
	at org.junit.Assert.fail(Assert.java:89)
	at org.junit.Assert.assertTrue(Assert.java:42)
	at org.junit.Assert.assertNotNull(Assert.java:713)
	at org.apache.solr.cloud.ReindexCollectionTest.testAbort(ReindexCollectionTest.java:425)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 6633 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22111ms
ok 43 post solr log into solr in 8653ms
ok 44 start solr with ssl in 20020ms
ok 45 start solr with ssl and auth in 23084ms
ok 46 SOLR-11740 check 'solr stop' connection in 28026ms
ok 47 status detects locally running solr in 14373ms
ok 48 status does not expose cli parameters to end user in 207ms
ok 49 listing out files in 2140ms
ok 50 copying files around in 6486ms
ok 51 upconfig in 2591ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.ReindexCollectionTest.testAbort (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.ReindexCollectionTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.ReindexCollectionTest.testAbort" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A989B10F32D4C3D9 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 56s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m22.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4758 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4758/

All tests passed

Build Log:
[...truncated 1254 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22277ms
ok 43 post solr log into solr in 9140ms
ok 44 start solr with ssl in 19774ms
ok 45 start solr with ssl and auth in 22673ms
ok 46 SOLR-11740 check 'solr stop' connection in 28346ms
ok 47 status detects locally running solr in 14644ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2159ms
ok 50 copying files around in 6529ms
ok 51 upconfig in 2608ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m23.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4757 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4757/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22721ms
ok 43 post solr log into solr in 9234ms
ok 44 start solr with ssl in 19762ms
ok 45 start solr with ssl and auth in 22313ms
ok 46 SOLR-11740 check 'solr stop' connection in 28417ms
ok 47 status detects locally running solr in 14592ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2157ms
ok 50 copying files around in 6475ms
ok 51 upconfig in 2600ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 54s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4756 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4756/

3 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8:4B972080FE1432E8]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 84.32 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more


FAILED:  org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8]:0)


FAILED:  org.apache.solr.client.solrj.TestLBHttpSolrClient.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8]:0)




Build Log:
[...truncated 1644 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22868ms
ok 43 post solr log into solr in 9188ms
ok 44 start solr with ssl in 19927ms
ok 45 start solr with ssl and auth in 22516ms
ok 46 SOLR-11740 check 'solr stop' connection in 28441ms
ok 47 status detects locally running solr in 14398ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6599ms
ok 51 upconfig in 2608ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:solrj:test

org.apache.solr.client.solrj.TestLBHttpSolrClient > testSimple FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8]:0)

org.apache.solr.client.solrj.TestLBHttpSolrClient > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8]:0)

org.apache.solr.client.solrj.TestLBHttpSolrClient > test suite's output saved to /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttpSolrClient.txt, copied below:
  2> 2723 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 2759 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 2806 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/data-dir-1-001
  2> 2903 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=false
  2> 2940 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 2969 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 3001 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.SolrTestCaseJ4 ###Starting testSimple
  2> 3564 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 3566 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 3574 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 3658 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 3713 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@383ba9e2{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,AVAILABLE}
  2> 3730 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.AbstractConnector Started ServerConnector@115d571a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37213}
  2> 3741 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.Server Started Server@5a3b81d0{STARTING}[10.0.15,sto=0] @3764ms
  2> 3752 ERROR (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 3753 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 3763 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]
  2> 3764 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 3764 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 3765 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-06-30T22:15:19.259053Z
  2> 3767 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001 (source: servlet config: solr.solr.home)
  2> 3785 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001/solr.xml
  2> 5291 WARN  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 5578 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001
  2> 5580 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 5617 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 5628 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.6.0
  2> 5893 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=test
  2> 5910 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.D.s.TrieIntField Solr loaded a deprecated plugin/analysis class [solr.TrieIntField]. Please consult documentation how to replace it accordingly.
  2> 5927 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema test/1.2 with uniqueid field id
  2> 5927 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001/collection1, trusted=true
  2> 5989 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001/collection1], dataDir=[/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001/collection1/data/]
  2> 6651 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 6761 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 6813 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 6813 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 6855 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, solrconfig=bad_solrconfig.xml, solr.data.dir=/tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001/collection1/data, hostPort=37213}
  2> 6943 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001/instance-0-001/collection1/conf
  2> 6962 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 6980 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> jul. 01, 2023 2:15:17 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> ADVERTENCIA: Suite execution timed out: org.apache.solr.client.solrj.TestLBHttpSolrClient
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@ef00653
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@ef00653
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@30b2b62c
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@30b2b62c
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@5b943e94
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@5b943e94
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:36864 to /127.0.0.1:46455 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7cc7e419
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7cc7e419
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@3a5d0960
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.stop(TestWorker.java:189)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@62150f9e
  2> 
  2> "/127.0.0.1:36864 to /127.0.0.1:46455 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59612e7b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59612e7b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "/127.0.0.1:36864 to /127.0.0.1:46455 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@23fbdb5d
  2> 	- locked sun.nio.ch.EPollSelectorImpl@713fd04b
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@409bf450
  2> 
  2> "SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@321b1559
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@675cafb9
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@675cafb9
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@6259e797
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@1f05e9b2
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.client.solrj.TestLBHttpSolrClient$SolrInstance.startJetty(TestLBHttpSolrClient.java:330)
  2> 	at app//org.apache.solr.client.solrj.TestLBHttpSolrClient.setUp(TestLBHttpSolrClient.java:89)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021bc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1037cf91
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7aeca113
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@57a49486
  2> 
  2> "Connection evictor" ID=16 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-18" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@49b9bf84
  2> 	- locked sun.nio.ch.EPollSelectorImpl@56426b99
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-19-acceptor-0@39eaf3ec-ServerConnector@115d571a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37213}" ID=19 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2531c573
  2> 
  2> "qtp1981821380-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1981821380-25" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2712870b
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-6d35b4c2-1" ID=26 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23d359f2
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@23d359f2
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6688f0b2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4d39ab4d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@441b5b42
  2> 
  2> "httpShardExecutor-6-thread-2" ID=28 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7ea0573e
  2> 	- locked sun.nio.ch.EPollSelectorImpl@47859dd1
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@64e92783
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=30 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6629e8f3
  2> 	- locked sun.nio.ch.EPollSelectorImpl@66ca44b8
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@2e5decd8
  2> 
  2> "h2sc-8-thread-2" ID=32 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1a2fe6f8
  2> 	- locked sun.nio.ch.EPollSelectorImpl@71af9b79
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@b1c22cd
  2> 
  2> "CloserThread" ID=34 WAITING on java.lang.Object@58e89bcf
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@58e89bcf
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=37 WAITING on java.util.concurrent.ForkJoinPool@6cb3ebee
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@6cb3ebee
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=38 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9ced8a6
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@9ced8a6
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8]:0)
  2> NOTE: reproduce with: gradlew test --tests TestLBHttpSolrClient.testSimple -Dtests.seed=A34B8D620DFC28C8 -Dtests.locale=es-CL -Dtests.timezone=Libya -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> jul. 01, 2023 2:15:17 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> ADVERTENCIA: Will linger awaiting termination of 16 leaked thread(s).
  2> jul. 01, 2023 2:15:18 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> GRAVE: 16 threads leaked from SUITE scope at org.apache.solr.client.solrj.TestLBHttpSolrClient: 
  2>    1) Thread[id=21, name=qtp1981821380-21, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=19, name=qtp1981821380-19-acceptor-0@39eaf3ec-ServerConnector@115d571a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37213}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=22, name=qtp1981821380-22, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=24, name=qtp1981821380-24, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=34, name=CloserThread, state=WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>    6) Thread[id=20, name=qtp1981821380-20, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=27, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=18, name=qtp1981821380-18, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=15, name=TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8], state=WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.client.solrj.TestLBHttpSolrClient$SolrInstance.startJetty(TestLBHttpSolrClient.java:330)
  2>         at app//org.apache.solr.client.solrj.TestLBHttpSolrClient.setUp(TestLBHttpSolrClient.java:89)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021bc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=25, name=qtp1981821380-25, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=23, name=qtp1981821380-23, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=31, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=26, name=Session-HouseKeeper-6d35b4c2-1, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=32, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=38, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=28, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x000000084034dc40.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x000000084042d840.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> jul. 01, 2023 2:15:18 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFORMACIÓN: Starting to interrupt leaked threads:
  2>    1) Thread[id=21, name=qtp1981821380-21, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>    2) Thread[id=19, name=qtp1981821380-19-acceptor-0@39eaf3ec-ServerConnector@115d571a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:37213}, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>    3) Thread[id=22, name=qtp1981821380-22, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>    4) Thread[id=24, name=qtp1981821380-24, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>    5) Thread[id=34, name=CloserThread, state=WAITING, group=TGRP-TestLBHttpSolrClient]
  2>    6) Thread[id=20, name=qtp1981821380-20, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>    7) Thread[id=27, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>    8) Thread[id=18, name=qtp1981821380-18, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>    9) Thread[id=15, name=TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8], state=WAITING, group=TGRP-TestLBHttpSolrClient]
  2>   10) Thread[id=25, name=qtp1981821380-25, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>   11) Thread[id=23, name=qtp1981821380-23, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>   12) Thread[id=31, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>   13) Thread[id=26, name=Session-HouseKeeper-6d35b4c2-1, state=TIMED_WAITING, group=TGRP-TestLBHttpSolrClient]
  2>   14) Thread[id=32, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2>   15) Thread[id=38, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestLBHttpSolrClient]
  2>   16) Thread[id=28, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestLBHttpSolrClient]
  2> 7203501 ERROR (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter.
  2>           => org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556)
  2> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 4a2a61a083fe325ff3f0fcbf7eb292909903a703 [snapshot build, details omitted]]
  2> 	at org.apache.solr.client.solrj.TestLBHttpSolrClient$SolrInstance.startJetty(TestLBHttpSolrClient.java:330) [test/:?]
  2> 	at org.apache.solr.client.solrj.TestLBHttpSolrClient.setUp(TestLBHttpSolrClient.java:89) [test/:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  2> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  2> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e -

[...truncated too long message...]

.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7204003 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.Server Stopped Server@5a3b81d0{STOPPING}[10.0.15,sto=0]
  2> 7204006 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.AbstractConnector Stopped ServerConnector@115d571a{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 7204009 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@383ba9e2{/solr,file:///tmp/src/solr/solr/solrj/build/tmp/tests-cwd/,STOPPED}
  2> 7204014 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1494334995
  2> 7204016 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@48af4bce collection1
  2> 7204017 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@48af4bce
  2> 7204043 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 7204063 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7204072 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7204077 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7204093 INFO  (TEST-TestLBHttpSolrClient.testSimple-seed#[A34B8D620DFC28C8]) [] o.a.s.SolrTestCaseJ4 ###Ending testSimple
  2> 7204097 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 7204097 INFO  (SUITE-TestLBHttpSolrClient-seed#[A34B8D620DFC28C8]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/solrj/build/tmp/tests-tmp/solr.client.solrj.TestLBHttpSolrClient_A34B8D620DFC28C8-001
  2> Jul 01, 2023 12:15:19 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([A34B8D620DFC28C8]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95), sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=es-CL, timezone=Libya
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=267574656,total=324534272
  2> NOTE: All tests run in this JVM: [TestLBHttpSolrClient]
  2> NOTE: reproduce with: gradlew test --tests TestLBHttpSolrClient -Dtests.seed=A34B8D620DFC28C8 -Dtests.locale=es-CL -Dtests.timezone=Libya -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:solrj:test (FAILURE): 1133 test(s), 2 failure(s), 40 skipped

1133 tests completed, 2 failed, 40 skipped

> Task :solr:solrj:test FAILED
> Task :solr:solrj:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttpSolrClient.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.TestLBHttpSolrClient.testSimple" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A34B8D620DFC28C8 -Ptests.file.encoding=US-ASCII

  - org.apache.solr.client.solrj.TestLBHttpSolrClient.classMethod (:solr:solrj)
    Test output: /tmp/src/solr/solr/solrj/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.TestLBHttpSolrClient.txt
    Reproduce with: gradlew :solr:solrj:test --tests "org.apache.solr.client.solrj.TestLBHttpSolrClient.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A34B8D620DFC28C8 -Ptests.file.encoding=US-ASCII

  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A34B8D620DFC28C8 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 3 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

3: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:solrj:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/solrj/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 41s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h3m7s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4755 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4755/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22782ms
ok 43 post solr log into solr in 8819ms
ok 44 start solr with ssl in 20181ms
ok 45 start solr with ssl and auth in 22178ms
ok 46 SOLR-11740 check 'solr stop' connection in 28018ms
ok 47 status detects locally running solr in 14673ms
ok 48 status does not expose cli parameters to end user in 208ms
ok 49 listing out files in 2184ms
ok 50 copying files around in 6531ms
ok 51 upconfig in 2670ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 50s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m16.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4754 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4754/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22644ms
ok 43 post solr log into solr in 9107ms
ok 44 start solr with ssl in 19838ms
ok 45 start solr with ssl and auth in 22597ms
ok 46 SOLR-11740 check 'solr stop' connection in 28503ms
ok 47 status detects locally running solr in 14710ms
ok 48 status does not expose cli parameters to end user in 216ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6572ms
ok 51 upconfig in 2646ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 2s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4753 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4753/

All tests passed

Build Log:
[...truncated 1241 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23284ms
ok 43 post solr log into solr in 9822ms
ok 44 start solr with ssl in 20730ms
ok 45 start solr with ssl and auth in 23568ms
ok 46 SOLR-11740 check 'solr stop' connection in 28617ms
ok 47 status detects locally running solr in 14850ms
ok 48 status does not expose cli parameters to end user in 233ms
ok 49 listing out files in 2266ms
ok 50 copying files around in 6943ms
ok 51 upconfig in 2875ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 48s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 17m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4752 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4752/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([D7CED3BA50897D35:3F127E58A3616715]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 80.66 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1655 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22555ms
ok 43 post solr log into solr in 8940ms
ok 44 start solr with ssl in 20075ms
ok 45 start solr with ssl and auth in 22919ms
ok 46 SOLR-11740 check 'solr stop' connection in 28366ms
ok 47 status detects locally running solr in 14378ms
ok 48 status does not expose cli parameters to end user in 221ms
ok 49 listing out files in 2155ms
ok 50 copying files around in 6528ms
ok 51 upconfig in 2621ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=D7CED3BA50897D35 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 53s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m18.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4751 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4751/

All tests passed

Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22286ms
ok 43 post solr log into solr in 9025ms
ok 44 start solr with ssl in 20204ms
ok 45 start solr with ssl and auth in 22511ms
ok 46 SOLR-11740 check 'solr stop' connection in 28021ms
ok 47 status detects locally running solr in 14349ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2184ms
ok 50 copying files around in 6598ms
ok 51 upconfig in 2624ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4750 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4750/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22706ms
ok 43 post solr log into solr in 9399ms
ok 44 start solr with ssl in 20212ms
ok 45 start solr with ssl and auth in 22689ms
ok 46 SOLR-11740 check 'solr stop' connection in 28061ms
ok 47 status detects locally running solr in 14348ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2144ms
ok 50 copying files around in 6503ms
ok 51 upconfig in 2620ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 5s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m38.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4749 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4749/

1 tests failed.
FAILED:  org.apache.solr.client.solrj.io.stream.MathExpressionTest.testZipFDistribution

Error Message:
java.lang.Exception: Zipf distribution not descending!!!

Stack Trace:
java.lang.Exception: Zipf distribution not descending!!!
	at __randomizedtesting.SeedInfo.seed([F965625B08F50846:DDD00F7F1F5D006E]:0)
	at org.apache.solr.client.solrj.io.stream.MathExpressionTest.testZipFDistribution(MathExpressionTest.java:4052)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)




Build Log:
[...truncated 1254 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.client.solrj.io.stream.MathExpressionTest.testZipFDistribution (:solr:solrj-streaming)
    Test output: /tmp/src/solr/solr/solrj-streaming/build/test-results/test/outputs/OUTPUT-org.apache.solr.client.solrj.io.stream.MathExpressionTest.txt
    Reproduce with: gradlew :solr:solrj-streaming:test --tests "org.apache.solr.client.solrj.io.stream.MathExpressionTest.testZipFDistribution" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=F965625B08F50846 -Ptests.file.encoding=US-ASCII


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':solr:solrj-streaming:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/solrj-streaming/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 5m 42s
616 actionable tasks: 600 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 6m7s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4748 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4748/

All tests passed

Build Log:
[...truncated 1247 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22733ms
ok 43 post solr log into solr in 8847ms
ok 44 start solr with ssl in 20274ms
ok 45 start solr with ssl and auth in 22027ms
ok 46 SOLR-11740 check 'solr stop' connection in 28044ms
ok 47 status detects locally running solr in 14366ms
ok 48 status does not expose cli parameters to end user in 206ms
ok 49 listing out files in 2154ms
ok 50 copying files around in 6506ms
ok 51 upconfig in 2674ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m22.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4747 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4747/

All tests passed

Build Log:
[...truncated 1258 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22554ms
ok 43 post solr log into solr in 8869ms
ok 44 start solr with ssl in 20437ms
ok 45 start solr with ssl and auth in 22366ms
ok 46 SOLR-11740 check 'solr stop' connection in 28599ms
ok 47 status detects locally running solr in 14362ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2170ms
ok 50 copying files around in 6549ms
ok 51 upconfig in 2643ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m25s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4746 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4746/

All tests passed

Build Log:
[...truncated 1252 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22888ms
ok 43 post solr log into solr in 9454ms
ok 44 start solr with ssl in 19750ms
ok 45 start solr with ssl and auth in 22291ms
ok 46 SOLR-11740 check 'solr stop' connection in 28161ms
ok 47 status detects locally running solr in 14680ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2174ms
ok 50 copying files around in 6590ms
ok 51 upconfig in 2658ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4745 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4745/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([93CD199BD5E63986:7B11B479260E23A6]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 88.81 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1655 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22529ms
ok 43 post solr log into solr in 9140ms
ok 44 start solr with ssl in 20165ms
ok 45 start solr with ssl and auth in 22299ms
ok 46 SOLR-11740 check 'solr stop' connection in 28686ms
ok 47 status detects locally running solr in 14332ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2163ms
ok 50 copying files around in 6515ms
ok 51 upconfig in 2670ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=93CD199BD5E63986 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 56s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m21.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4744 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4744/

1 tests failed.
FAILED:  org.apache.solr.ltr.feature.TestExternalFeatures.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([EACED88D8A25002C]:0)




Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23216ms
ok 43 post solr log into solr in 9251ms
ok 44 start solr with ssl in 20151ms
ok 45 start solr with ssl and auth in 22422ms
ok 46 SOLR-11740 check 'solr stop' connection in 28367ms
ok 47 status detects locally running solr in 14373ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2152ms
ok 50 copying files around in 6507ms
ok 51 upconfig in 2590ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.feature.TestExternalFeatures > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([EACED88D8A25002C]:0)

org.apache.solr.ltr.feature.TestExternalFeatures > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestExternalFeatures.txt, copied below:
  1> 16:58:35.550 [SUITE-TestExternalFeatures-seed#[EACED88D8A25002C]-worker] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> Jun 29, 2023 9:58:33 PM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.ltr.feature.TestExternalFeatures
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@6deec08a
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@6deec08a
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@2e39e131
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@2e39e131
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@22db988b
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@22db988b
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:50260 to /127.0.0.1:41453 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c3299fd
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2c3299fd
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:50260 to /127.0.0.1:41453 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d054e11
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5d054e11
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:50260 to /127.0.0.1:41453 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3c541ce9
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4235c428
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestExternalFeatures-seed#[EACED88D8A25002C]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@7af040e2
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "SUITE-TestExternalFeatures-seed#[EACED88D8A25002C]-worker" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@66d6c245
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@66d6c245
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@df2fc6b
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@11dd1a68
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.feature.TestExternalFeatures.before(TestExternalFeatures.java:30)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7d2598ff
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@52a30e9
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1fa19583
  2> 
  2> "qtp659313408-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@37654c00
  2> 	- locked sun.nio.ch.EPollSelectorImpl@631c9f7d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp659313408-18-acceptor-0@419b68fc-ServerConnector@5dcc6157{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:35491}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@70020310
  2> 
  2> "qtp659313408-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp659313408-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp659313408-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp659313408-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp659313408-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp659313408-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@201633bf
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-5473705d-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d0e7ef1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3d0e7ef1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3aeee95a
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5a147794
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@46ba069e
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4b0e2ca2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5b92ac4
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@2e88c58f
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4850b710
  2> 	- locked sun.nio.ch.EPollSelectorImpl@3181c159
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@6e4e0021
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@bd090f7
  2> 	- locked sun.nio.ch.EPollSelectorImpl@47f37443
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@48ea709c
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@ad6b39a
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@ad6b39a
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@35623090
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@35623090
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3aba49e3
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3aba49e3
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
  2> Jun 29, 2023 9:58:34 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> Jun 29, 2023 9:58:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.feature.TestExternalFeatures: 
  2>    1) Thread[id=19, name=qtp659313408-19, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=24, name=qtp659313408-24, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=20, name=qtp659313408-20, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=25, name=Session-HouseKeeper-5473705d-1, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=18, name=qtp659313408-18-acceptor-0@419b68fc-ServerConnector@5dcc6157{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:35491}, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=15, name=SUITE-TestExternalFeatures-seed#[EACED88D8A25002C]-worker, state=WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.feature.TestExternalFeatures.before(TestExternalFeatures.java:30)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2564)
  2>   11) Thread[id=21, name=qtp659313408-21, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$344/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=17, name=qtp659313408-17, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$253/0x0000000840305040.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=22, name=qtp659313408-22, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=23, name=qtp659313408-23, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> Jun 29, 2023 9:58:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=19, name=qtp659313408-19, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>    2) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>    3) Thread[id=24, name=qtp659313408-24, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>    4) Thread[id=20, name=qtp659313408-20, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>    5) Thread[id=25, name=Session-HouseKeeper-5473705d-1, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>    6) Thread[id=18, name=qtp659313408-18-acceptor-0@419b68fc-ServerConnector@5dcc6157{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:35491}, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>    7) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>    8) Thread[id=15, name=SUITE-TestExternalFeatures-seed#[EACED88D8A25002C]-worker, state=WAITING, group=TGRP-TestExternalFeatures]
  2>    9) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>   10) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestExternalFeatures]
  2>   11) Thread[id=21, name=qtp659313408-21, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>   12) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestExternalFeatures]
  2>   13) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>   14) Thread[id=17, name=qtp659313408-17, state=RUNNABLE, group=TGRP-TestExternalFeatures]
  2>   15) Thread[id=22, name=qtp659313408-22, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  2>   16) Thread[id=23, name=qtp659313408-23, state=TIMED_WAITING, group=TGRP-TestExternalFeatures]
  1> 18:58:35.054 [SUITE-TestExternalFeatures-seed#[EACED88D8A25002C]-worker] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT e607f6a00bce23538a92848f3306ade292ce7f5d [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.feature.TestExternalFeatures.before(TestExternalFeatures.java:30) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> Jun 29, 2023 6:58:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([EACED88D8A25002C]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=269, maxMBSortInHeap=7.712784742694083, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=en-US-POSIX, timezone=Europe/Volgograd
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=255867968,total=324534272
  2> NOTE: All tests run in this JVM: [TestExternalFeatures]
  2> NOTE: reproduce with: gradlew test --tests TestExternalFeatures -Dtests.seed=EACED88D8A25002C -Dtests.locale=en-US-POSIX -Dtests.timezone=Europe/Volgograd -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 180 test(s), 1 failure(s)

180 tests completed, 1 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.feature.TestExternalFeatures.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.feature.TestExternalFeatures.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.feature.TestExternalFeatures.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=EACED88D8A25002C -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 24s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m51s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4743 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4743/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22301ms
ok 43 post solr log into solr in 9153ms
ok 44 start solr with ssl in 19694ms
ok 45 start solr with ssl and auth in 23159ms
ok 46 SOLR-11740 check 'solr stop' connection in 28052ms
ok 47 status detects locally running solr in 14379ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2162ms
ok 50 copying files around in 6517ms
ok 51 upconfig in 2629ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m25s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4742 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4742/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([A4C56364B5D49D93:4C19CE86463C87B3]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 88.27 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1656 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22265ms
ok 43 post solr log into solr in 8850ms
ok 44 start solr with ssl in 20498ms
ok 45 start solr with ssl and auth in 22763ms
ok 46 SOLR-11740 check 'solr stop' connection in 28307ms
ok 47 status detects locally running solr in 14391ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2163ms
ok 50 copying files around in 6646ms
ok 51 upconfig in 2607ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=A4C56364B5D49D93 -Ptests.file.encoding=ISO-8859-1


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 4s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m29.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4741 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4741/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22688ms
ok 43 post solr log into solr in 10047ms
ok 44 start solr with ssl in 20475ms
ok 45 start solr with ssl and auth in 22830ms
ok 46 SOLR-11740 check 'solr stop' connection in 28114ms
ok 47 status detects locally running solr in 14425ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6571ms
ok 51 upconfig in 2604ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 56s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m20s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4740 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4740/

All tests passed

Build Log:
[...truncated 1243 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22655ms
ok 43 post solr log into solr in 8993ms
ok 44 start solr with ssl in 19993ms
ok 45 start solr with ssl and auth in 22764ms
ok 46 SOLR-11740 check 'solr stop' connection in 28729ms
ok 47 status detects locally running solr in 14377ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6632ms
ok 51 upconfig in 2654ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m21.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4739 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4739/

All tests passed

Build Log:
[...truncated 1251 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22634ms
ok 43 post solr log into solr in 9422ms
ok 44 start solr with ssl in 19934ms
ok 45 start solr with ssl and auth in 22741ms
ok 46 SOLR-11740 check 'solr stop' connection in 28494ms
ok 47 status detects locally running solr in 14432ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2178ms
ok 50 copying files around in 6604ms
ok 51 upconfig in 2659ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m21.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4738 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4738/

1 tests failed.
FAILED:  org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestLeaderElectionZkExpiry: 
   1) Thread[id=252, name=zkConnectionManagerCallback-160-thread-1-EventThread, state=BLOCKED, group=TGRP-TestLeaderElectionZkExpiry]
        at app//org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:558)
        at app//org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:440)
        at app//org.apache.solr.cloud.ZkController.lambda$new$0(ZkController.java:386)
        at app//org.apache.solr.cloud.ZkController$$Lambda$430/0x0000000840462440.command(Unknown Source)
        at app//org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:199)
        at app//org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:59)
        at app//org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:179)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:578)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553)
   2) Thread[id=250, name=zkConnectionManagerCallback-160-thread-1-EventThread, state=BLOCKED, group=TGRP-TestLeaderElectionZkExpiry]
        at app//org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:558)
        at app//org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:440)
        at app//org.apache.solr.cloud.ZkController.lambda$new$0(ZkController.java:386)
        at app//org.apache.solr.cloud.ZkController$$Lambda$430/0x0000000840462440.command(Unknown Source)
        at app//org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:199)
        at app//org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:59)
        at app//org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:179)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:578)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553)
   3) Thread[id=259, name=OverseerExitThread, state=TIMED_WAITING, group=Overseer state updater.]
        at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
        at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:161)
        at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:82)
        at app//org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:401)
        at app//org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
        at app//org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:395)
        at app//org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:138)
        at app//org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:315)
        at app//org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:395)
        at app//org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2435)
        at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:511)
        at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater$$Lambda$1247/0x0000000840922c40.run(Unknown Source)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 3 threads leaked from SUITE scope at org.apache.solr.cloud.TestLeaderElectionZkExpiry: 
   1) Thread[id=252, name=zkConnectionManagerCallback-160-thread-1-EventThread, state=BLOCKED, group=TGRP-TestLeaderElectionZkExpiry]
        at app//org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:558)
        at app//org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:440)
        at app//org.apache.solr.cloud.ZkController.lambda$new$0(ZkController.java:386)
        at app//org.apache.solr.cloud.ZkController$$Lambda$430/0x0000000840462440.command(Unknown Source)
        at app//org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:199)
        at app//org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:59)
        at app//org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:179)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:578)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553)
   2) Thread[id=250, name=zkConnectionManagerCallback-160-thread-1-EventThread, state=BLOCKED, group=TGRP-TestLeaderElectionZkExpiry]
        at app//org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:558)
        at app//org.apache.solr.cloud.ZkController.onReconnect(ZkController.java:440)
        at app//org.apache.solr.cloud.ZkController.lambda$new$0(ZkController.java:386)
        at app//org.apache.solr.cloud.ZkController$$Lambda$430/0x0000000840462440.command(Unknown Source)
        at app//org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:199)
        at app//org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:59)
        at app//org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:179)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:578)
        at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:553)
   3) Thread[id=259, name=OverseerExitThread, state=TIMED_WAITING, group=Overseer state updater.]
        at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
        at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:161)
        at app//org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:82)
        at app//org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:401)
        at app//org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:277)
        at app//org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:395)
        at app//org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:138)
        at app//org.apache.solr.cloud.LeaderElector.joinElection(LeaderElector.java:315)
        at app//org.apache.solr.cloud.LeaderElector.retryElection(LeaderElector.java:395)
        at app//org.apache.solr.cloud.ZkController.rejoinOverseerElection(ZkController.java:2435)
        at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater.checkIfIamStillLeader(Overseer.java:511)
        at app//org.apache.solr.cloud.Overseer$ClusterStateUpdater$$Lambda$1247/0x0000000840922c40.run(Unknown Source)
        at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
	at __randomizedtesting.SeedInfo.seed([20FBFA35B531D2FF]:0)




Build Log:
[...truncated 2419 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22324ms
ok 43 post solr log into solr in 9305ms
ok 44 start solr with ssl in 20157ms
ok 45 start solr with ssl and auth in 22352ms
ok 46 SOLR-11740 check 'solr stop' connection in 28407ms
ok 47 status detects locally running solr in 14394ms
ok 48 status does not expose cli parameters to end user in 213ms
ok 49 listing out files in 2184ms
ok 50 copying files around in 6541ms
ok 51 upconfig in 2672ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.TestLeaderElectionZkExpiry.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.TestLeaderElectionZkExpiry.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=20FBFA35B531D2FF -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 57s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m19.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4737 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4737/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22195ms
ok 43 post solr log into solr in 9309ms
ok 44 start solr with ssl in 20135ms
ok 45 start solr with ssl and auth in 22584ms
ok 46 SOLR-11740 check 'solr stop' connection in 28141ms
ok 47 status detects locally running solr in 14371ms
ok 48 status does not expose cli parameters to end user in 214ms
ok 49 listing out files in 2239ms
ok 50 copying files around in 6560ms
ok 51 upconfig in 2697ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 56s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m17s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4736 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4736/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22222ms
ok 43 post solr log into solr in 8930ms
ok 44 start solr with ssl in 20444ms
ok 45 start solr with ssl and auth in 22375ms
ok 46 SOLR-11740 check 'solr stop' connection in 28364ms
ok 47 status detects locally running solr in 14677ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2179ms
ok 50 copying files around in 6525ms
ok 51 upconfig in 2602ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m25.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4735 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4735/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([5AF911187FFAA188:B225BCFA8C12BBA8]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 84.89 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1645 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22331ms
ok 43 post solr log into solr in 9229ms
ok 44 start solr with ssl in 20436ms
ok 45 start solr with ssl and auth in 22684ms
ok 46 SOLR-11740 check 'solr stop' connection in 28379ms
ok 47 status detects locally running solr in 14338ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2163ms
ok 50 copying files around in 6615ms
ok 51 upconfig in 2602ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=5AF911187FFAA188 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 16m
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m35s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4734 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4734/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([529AD549518E68CF:BA4678ABA26672EF]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 106.17 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1657 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23144ms
ok 43 post solr log into solr in 9267ms
ok 44 start solr with ssl in 20466ms
ok 45 start solr with ssl and auth in 22511ms
ok 46 SOLR-11740 check 'solr stop' connection in 28363ms
ok 47 status detects locally running solr in 14365ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2205ms
ok 50 copying files around in 6562ms
ok 51 upconfig in 2720ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=529AD549518E68CF -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 16s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m48s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4733 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4733/

2 tests failed.
FAILED:  org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight

Error Message:
java.lang.Exception: Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([D64483E7B56CD724]:0)


FAILED:  org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([D64483E7B56CD724]:0)




Build Log:
[...truncated 1244 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22743ms
ok 43 post solr log into solr in 8843ms
ok 44 start solr with ssl in 20056ms
ok 45 start solr with ssl and auth in 21956ms
ok 46 SOLR-11740 check 'solr stop' connection in 28343ms
ok 47 status detects locally running solr in 14429ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2160ms
ok 50 copying files around in 6536ms
ok 51 upconfig in 2596ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel > multipleAdditiveTreesTestNoRight FAILED
    java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([D64483E7B56CD724]:0)

org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([D64483E7B56CD724]:0)

org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.txt, copied below:
  1> 12:10:36.215 [TEST-TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight-seed#[D64483E7B56CD724]] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> kor 28, 2023 8:10:34 KIKIIƊE com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@380cfdf9
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@380cfdf9
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@532e3361
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@532e3361
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@7130b224
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@7130b224
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:34562 to /127.0.0.1:33417 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1ff771
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7f1ff771
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:34562 to /127.0.0.1:33417 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fc044f2
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1fc044f2
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:34562 to /127.0.0.1:33417 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@3d57d950
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4902fd4c
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestMultipleAdditiveTreesModel-seed#[D64483E7B56CD724]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@49a62f62
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "TEST-TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight-seed#[D64483E7B56CD724]" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@4ce75ec
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@4ce75ec
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@54dc1570
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@5742cbe2
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.before(TestMultipleAdditiveTreesModel.java:32)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@9f4a4db
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@715157b8
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4512ab77
  2> 
  2> "qtp1188379246-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@2830dbd2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@53a7a5fb
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1188379246-18-acceptor-0@17ef17f9-ServerConnector@1f4be046{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:45269}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@3bef5eb
  2> 
  2> "qtp1188379246-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1188379246-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1188379246-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1188379246-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1188379246-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1188379246-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@74a88165
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-5e31f6fe-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ae82a69
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5ae82a69
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@340ec4b2
  2> 	- locked sun.nio.ch.EPollSelectorImpl@40eb6e75
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@6997823f
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@70786ca4
  2> 	- locked sun.nio.ch.EPollSelectorImpl@6b58bba0
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@3ddfe3ff
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@29b1d92d
  2> 	- locked sun.nio.ch.EPollSelectorImpl@323282a8
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@4fcec496
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@19cd506
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2f37fb2e
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@f7054b5
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@8e82754
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@8e82754
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@2e69c323
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@2e69c323
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@493a070d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@493a070d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
   >     java.lang.Exception: Test abandoned because suite timeout was reached.
   >         at __randomizedtesting.SeedInfo.seed([D64483E7B56CD724]:0)
  2> NOTE: reproduce with: gradlew test --tests TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight -Dtests.seed=D64483E7B56CD724 -Dtests.locale=ff-MR -Dtests.timezone=Indian/Chagos -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> kor 28, 2023 8:10:34 KIKIIƊE com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> kor 28, 2023 8:10:35 KIKIIƊE com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel: 
  2>    1) Thread[id=15, name=TEST-TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight-seed#[D64483E7B56CD724], state=WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.before(TestMultipleAdditiveTreesModel.java:32)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=17, name=qtp1188379246-17, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=19, name=qtp1188379246-19, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=21, name=qtp1188379246-21, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=25, name=Session-HouseKeeper-5e31f6fe-1, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=22, name=qtp1188379246-22, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=20, name=qtp1188379246-20, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=18, name=qtp1188379246-18-acceptor-0@17ef17f9-ServerConnector@1f4be046{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:45269}, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=24, name=qtp1188379246-24, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$251/0x0000000840305040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$342/0x0000000840437c40.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=23, name=qtp1188379246-23, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2> kor 28, 2023 8:10:35 KIKIIƊE com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=15, name=TEST-TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight-seed#[D64483E7B56CD724], state=WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    2) Thread[id=17, name=qtp1188379246-17, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    3) Thread[id=19, name=qtp1188379246-19, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    4) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    5) Thread[id=21, name=qtp1188379246-21, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    6) Thread[id=25, name=Session-HouseKeeper-5e31f6fe-1, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    7) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    8) Thread[id=22, name=qtp1188379246-22, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>    9) Thread[id=20, name=qtp1188379246-20, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   10) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   11) Thread[id=18, name=qtp1188379246-18-acceptor-0@17ef17f9-ServerConnector@1f4be046{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:45269}, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   12) Thread[id=24, name=qtp1188379246-24, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   13) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   14) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   15) Thread[id=23, name=qtp1188379246-23, state=TIMED_WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  2>   16) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestMultipleAdditiveTreesModel]
  1> 14:10:35.684 [TEST-TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight-seed#[D64483E7B56CD724]] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT 608a5444c808ffc89b9ba9976a5aaf3c9521b6f4 [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.before(TestMultipleAdditiveTreesModel.java:32) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:980) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> NOTE: leaving temporary files on disk at: /tmp/src/solr/solr/modules/ltr/build/tmp/tests-tmp/solr.ltr.model.TestMultipleAdditiveTreesModel_D64483E7B56CD724-001
  2> Jun 28, 2023 2:10:36 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([D64483E7B56CD724]:0)
  2> NOTE: test params are: codec=DeflateWithPresetCompressingStoredFieldsData(storedFieldsFormat=Lucene90CompressingStoredFieldsFormat(compressionMode=BEST_COMPRESSION, chunkSize=28553, maxDocsPerChunk=4, blockShift=5), termVectorsFormat=Lucene90CompressingTermVectorsFormat(compressionMode=BEST_COMPRESSION, chunkSize=28553, maxDocsPerChunk=4, blockSize=5)), sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=ff-MR, timezone=Indian/Chagos
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=254912112,total=324534272
  2> NOTE: All tests run in this JVM: [TestMultipleAdditiveTreesModel]
  2> NOTE: reproduce with: gradlew test --tests TestMultipleAdditiveTreesModel -Dtests.seed=D64483E7B56CD724 -Dtests.locale=ff-MR -Dtests.timezone=Indian/Chagos -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 175 test(s), 2 failure(s)

175 tests completed, 2 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.multipleAdditiveTreesTestNoRight" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=D64483E7B56CD724 -Ptests.file.encoding=UTF-8

  - org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.model.TestMultipleAdditiveTreesModel.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=D64483E7B56CD724 -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 24s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h2m50s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4732 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4732/

All tests passed

Build Log:
[...truncated 1256 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22241ms
ok 43 post solr log into solr in 9109ms
ok 44 start solr with ssl in 19802ms
ok 45 start solr with ssl and auth in 22376ms
ok 46 SOLR-11740 check 'solr stop' connection in 28049ms
ok 47 status detects locally running solr in 14396ms
ok 48 status does not expose cli parameters to end user in 209ms
ok 49 listing out files in 2154ms
ok 50 copying files around in 6555ms
ok 51 upconfig in 2610ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 49s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m15s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4731 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4731/

All tests passed

Build Log:
[...truncated 1255 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22769ms
ok 43 post solr log into solr in 9059ms
ok 44 start solr with ssl in 19819ms
ok 45 start solr with ssl and auth in 22431ms
ok 46 SOLR-11740 check 'solr stop' connection in 28431ms
ok 47 status detects locally running solr in 14675ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2189ms
ok 50 copying files around in 6680ms
ok 51 upconfig in 2598ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 5s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m31.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4730 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4730/

All tests passed

Build Log:
[...truncated 1248 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22378ms
ok 43 post solr log into solr in 8813ms
ok 44 start solr with ssl in 19882ms
ok 45 start solr with ssl and auth in 22909ms
ok 46 SOLR-11740 check 'solr stop' connection in 28068ms
ok 47 status detects locally running solr in 14348ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2163ms
ok 50 copying files around in 6508ms
ok 51 upconfig in 2657ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 48s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m14s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4729 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4729/

All tests passed

Build Log:
[...truncated 1244 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22837ms
ok 43 post solr log into solr in 9589ms
ok 44 start solr with ssl in 20167ms
ok 45 start solr with ssl and auth in 22946ms
ok 46 SOLR-11740 check 'solr stop' connection in 28227ms
ok 47 status detects locally running solr in 14473ms
ok 48 status does not expose cli parameters to end user in 219ms
ok 49 listing out files in 2203ms
ok 50 copying files around in 6681ms
ok 51 upconfig in 2651ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 6s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m31s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4728 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4728/

All tests passed

Build Log:
[...truncated 1260 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22550ms
ok 43 post solr log into solr in 9174ms
ok 44 start solr with ssl in 19501ms
ok 45 start solr with ssl and auth in 22430ms
ok 46 SOLR-11740 check 'solr stop' connection in 28031ms
ok 47 status detects locally running solr in 14361ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2150ms
ok 50 copying files around in 6477ms
ok 51 upconfig in 2565ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 12m 58s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m25.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4727 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4727/

1 tests failed.
FAILED:  org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming

Error Message:
java.lang.RuntimeException: Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
	at __randomizedtesting.SeedInfo.seed([EFA5BC0AC54EE9E0:77911E836A6F3C0]:0)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:1024)
	at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:982)
	at org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming(TestCircuitBreaker.java:220)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.solr.common.SolrException: Circuit Breakers tripped CPU Circuit Breaker triggered as seen CPU usage is above allowed threshold.Seen CPU usage 89.5 and allocated threshold 75.0

	at app//org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:375)
	at app//org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:224)
	at app//org.apache.solr.core.SolrCore.execute(SolrCore.java:2893)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:348)
	at app//org.apache.solr.util.TestHarness.query(TestHarness.java:328)
	at app//org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:996)
	... 49 more




Build Log:
[...truncated 1648 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22694ms
ok 43 post solr log into solr in 8830ms
ok 44 start solr with ssl in 19855ms
ok 45 start solr with ssl and auth in 22142ms
ok 46 SOLR-11740 check 'solr stop' connection in 28267ms
ok 47 status detects locally running solr in 14338ms
ok 48 status does not expose cli parameters to end user in 205ms
ok 49 listing out files in 2146ms
ok 50 copying files around in 6496ms
ok 51 upconfig in 2571ms

> Task :solr:packaging:integrationTests FAILED

ERROR: The following test(s) have failed:
  - org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.util.TestCircuitBreaker.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.util.TestCircuitBreaker.testResponseWithCBTiming" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=EFA5BC0AC54EE9E0 -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 54s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m19s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4726 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4726/

All tests passed

Build Log:
[...truncated 1259 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 23076ms
ok 43 post solr log into solr in 8982ms
ok 44 start solr with ssl in 19786ms
ok 45 start solr with ssl and auth in 22489ms
ok 46 SOLR-11740 check 'solr stop' connection in 28071ms
ok 47 status detects locally running solr in 14417ms
ok 48 status does not expose cli parameters to end user in 212ms
ok 49 listing out files in 2172ms
ok 50 copying files around in 6673ms
ok 51 upconfig in 2638ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 10s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m36s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4725 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4725/

All tests passed

Build Log:
[...truncated 1250 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22139ms
ok 43 post solr log into solr in 9200ms
ok 44 start solr with ssl in 19951ms
ok 45 start solr with ssl and auth in 22311ms
ok 46 SOLR-11740 check 'solr stop' connection in 28573ms
ok 47 status detects locally running solr in 14314ms
ok 48 status does not expose cli parameters to end user in 206ms
ok 49 listing out files in 2153ms
ok 50 copying files around in 6522ms
ok 51 upconfig in 2608ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 51s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m15s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4724 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4724/

All tests passed

Build Log:
[...truncated 1257 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22566ms
ok 43 post solr log into solr in 9305ms
ok 44 start solr with ssl in 20245ms
ok 45 start solr with ssl and auth in 22230ms
ok 46 SOLR-11740 check 'solr stop' connection in 28174ms
ok 47 status detects locally running solr in 14438ms
ok 48 status does not expose cli parameters to end user in 215ms
ok 49 listing out files in 2190ms
ok 50 copying files around in 6578ms
ok 51 upconfig in 2726ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 13m 16s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 13m44.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4723 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4723/

All tests passed

Build Log:
[...truncated 1249 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22564ms
ok 43 post solr log into solr in 8807ms
ok 44 start solr with ssl in 20092ms
ok 45 start solr with ssl and auth in 22523ms
ok 46 SOLR-11740 check 'solr stop' connection in 28656ms
ok 47 status detects locally running solr in 14387ms
ok 48 status does not expose cli parameters to end user in 218ms
ok 49 listing out files in 2175ms
ok 50 copying files around in 6577ms
ok 51 upconfig in 2657ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 15m 57s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m32s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4722 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4722/

All tests passed

Build Log:
[...truncated 1252 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22442ms
ok 43 post solr log into solr in 9196ms
ok 44 start solr with ssl in 20190ms
ok 45 start solr with ssl and auth in 22255ms
ok 46 SOLR-11740 check 'solr stop' connection in 28047ms
ok 47 status detects locally running solr in 14351ms
ok 48 status does not expose cli parameters to end user in 210ms
ok 49 listing out files in 2165ms
ok 50 copying files around in 6512ms
ok 51 upconfig in 2606ms

> Task :solr:packaging:integrationTests FAILED

FAILURE: Build failed with an exception.

* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 16m 1s
644 actionable tasks: 628 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 16m26s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4721 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4721/

1 tests failed.
FAILED:  org.apache.solr.ltr.TestLTRQParserPlugin.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([B24231CABE1739BD]:0)




Build Log:
[...truncated 1242 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22766ms
ok 43 post solr log into solr in 9406ms
ok 44 start solr with ssl in 19731ms
ok 45 start solr with ssl and auth in 22310ms
ok 46 SOLR-11740 check 'solr stop' connection in 28396ms
ok 47 status detects locally running solr in 14414ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2222ms
ok 50 copying files around in 6647ms
ok 51 upconfig in 2656ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:modules:ltr:test

org.apache.solr.ltr.TestLTRQParserPlugin > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([B24231CABE1739BD]:0)

org.apache.solr.ltr.TestLTRQParserPlugin > test suite's output saved to /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTRQParserPlugin.txt, copied below:
  1> 07:40:16.722 [SUITE-TestLTRQParserPlugin-seed#[B24231CABE1739BD]-worker] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> jun. 27, 2023 4:40:14 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> ADVERTENCIA: Suite execution timed out: org.apache.solr.ltr.TestLTRQParserPlugin
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@59a5f3ce
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@59a5f3ce
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@75546e4a
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@75546e4a
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@13dc849d
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@13dc849d
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:41950 to /127.0.0.1:38675 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758de5c4
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@758de5c4
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:403)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42e99e4a
  2> 
  2> "/127.0.0.1:41950 to /127.0.0.1:38675 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@63d9424a
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@63d9424a
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@42a48628
  2> 
  2> "/127.0.0.1:41950 to /127.0.0.1:38675 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@35294645
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4226d411
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@74f0ea28
  2> 
  2> "SUITE-TestLTRQParserPlugin-seed#[B24231CABE1739BD]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@581faa54
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "SUITE-TestLTRQParserPlugin-seed#[B24231CABE1739BD]-worker" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@3e3f41d
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@3e3f41d
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@71fd780f
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@e99031b
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2> 	at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2> 	at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2> 	at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2> 	at app//org.apache.solr.ltr.TestLTRQParserPlugin.before(TestLTRQParserPlugin.java:28)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@7aebd885
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@5f038086
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@45a4b669
  2> 
  2> "qtp1599630796-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@27682b14
  2> 	- locked sun.nio.ch.EPollSelectorImpl@642c88cf
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1599630796-18-acceptor-0@621fd748-ServerConnector@593c3efe{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:42553}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@4db7ddd
  2> 
  2> "qtp1599630796-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1599630796-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1599630796-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1599630796-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1599630796-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp1599630796-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@18842247
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-519136f1-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e529ac8
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7e529ac8
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7eb81f3d
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2152b0a2
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@6d236fe3
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@18fd28d5
  2> 	- locked sun.nio.ch.EPollSelectorImpl@721365d8
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@73a9feb2
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4ceccd50
  2> 	- locked sun.nio.ch.EPollSelectorImpl@780fb3cf
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@29ce68f4
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@9c5879b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@60911ede
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@62ffe79a
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@4a3dd9d1
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@4a3dd9d1
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@4fa98a92
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@4fa98a92
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a7525a3
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6a7525a3
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
  2> jun. 27, 2023 4:40:14 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> ADVERTENCIA: Will linger awaiting termination of 16 leaked thread(s).
  2> jun. 27, 2023 4:40:15 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> GRAVE: 16 threads leaked from SUITE scope at org.apache.solr.ltr.TestLTRQParserPlugin: 
  2>    1) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=25, name=Session-HouseKeeper-519136f1-1, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=17, name=qtp1599630796-17, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2>    6) Thread[id=18, name=qtp1599630796-18-acceptor-0@621fd748-ServerConnector@593c3efe{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:42553}, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=20, name=qtp1599630796-20, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=23, name=qtp1599630796-23, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=22, name=qtp1599630796-22, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=15, name=SUITE-TestLTRQParserPlugin-seed#[B24231CABE1739BD]-worker, state=WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126)
  2>         at app//org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86)
  2>         at app//org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181)
  2>         at app//org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113)
  2>         at app//org.apache.solr.ltr.TestLTRQParserPlugin.before(TestLTRQParserPlugin.java:28)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$177/0x00000008401edc40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=21, name=qtp1599630796-21, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=24, name=qtp1599630796-24, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=19, name=qtp1599630796-19, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   16) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$257/0x000000084034d440.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$346/0x000000084044f440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> jun. 27, 2023 4:40:15 A. M. com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFORMACIÓN: Starting to interrupt leaked threads:
  2>    1) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>    2) Thread[id=25, name=Session-HouseKeeper-519136f1-1, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>    3) Thread[id=17, name=qtp1599630796-17, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>    4) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>    5) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestLTRQParserPlugin]
  2>    6) Thread[id=18, name=qtp1599630796-18-acceptor-0@621fd748-ServerConnector@593c3efe{ssl, (ssl, alpn, http/1.1, h2)}{127.0.0.1:42553}, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>    7) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  2>    8) Thread[id=20, name=qtp1599630796-20, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>    9) Thread[id=23, name=qtp1599630796-23, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   10) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   11) Thread[id=22, name=qtp1599630796-22, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   12) Thread[id=15, name=SUITE-TestLTRQParserPlugin-seed#[B24231CABE1739BD]-worker, state=WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   13) Thread[id=21, name=qtp1599630796-21, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   14) Thread[id=24, name=qtp1599630796-24, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   15) Thread[id=19, name=qtp1599630796-19, state=TIMED_WAITING, group=TGRP-TestLTRQParserPlugin]
  2>   16) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestLTRQParserPlugin]
  1> 09:40:15.508 [SUITE-TestLTRQParserPlugin-seed#[B24231CABE1739BD]-worker] ERROR org.apache.solr.servlet.SolrDispatchFilter - Could not start Dispatch Filter.
  1> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  1> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [solr-core-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  1> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:126) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.SolrJettyTestBase.createAndStartJetty(SolrJettyTestBase.java:86) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.util.RestTestBase.createJettyAndHarness(RestTestBase.java:57) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT d77d0c1b5fc1534ec6404570649fc641c27de331 [snapshot build, details omitted]]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:181) [test/:?]
  1> 	at org.apache.solr.ltr.TestRerankBase.setuptest(TestRerankBase.java:113) [test/:?]
  1> 	at org.apache.solr.ltr.TestLTRQParserPlugin.before(TestLTRQParserPlugin.java:28) [test/:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  1> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  1> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  1> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  1> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  1> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  1> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> Jun 27, 2023 9:40:16 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([B24231CABE1739BD]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=689, maxMBSortInHeap=5.24786376215366, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=es-EA, timezone=Jamaica
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=256388648,total=324534272
  2> NOTE: All tests run in this JVM: [TestLTRQParserPlugin]
  2> NOTE: reproduce with: gradlew test --tests TestLTRQParserPlugin -Dtests.seed=B24231CABE1739BD -Dtests.locale=es-EA -Dtests.timezone=Jamaica -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:modules:ltr:test (FAILURE): 181 test(s), 1 failure(s)

181 tests completed, 1 failed

> Task :solr:modules:ltr:test FAILED
> Task :solr:modules:ltr:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.ltr.TestLTRQParserPlugin.classMethod (:solr:modules:ltr)
    Test output: /tmp/src/solr/solr/modules/ltr/build/test-results/test/outputs/OUTPUT-org.apache.solr.ltr.TestLTRQParserPlugin.txt
    Reproduce with: gradlew :solr:modules:ltr:test --tests "org.apache.solr.ltr.TestLTRQParserPlugin.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=B24231CABE1739BD -Ptests.file.encoding=UTF-8


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:modules:ltr:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/modules/ltr/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 2m 47s
642 actionable tasks: 626 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h3m11.5s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1684) [Receiver=3.1.3]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

[JENKINS] Solr » Solr-Check-9.x - Build # 4720 - Still Failing!

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://ci-builds.apache.org/job/Solr/job/Solr-Check-9.x/4720/

1 tests failed.
FAILED:  org.apache.solr.TestSolrCoreProperties.classMethod

Error Message:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([405BE1D6CAE1C16E]:0)




Build Log:
[...truncated 1225 lines...]
# ERROR: Error from server at http://localhost:8983/solr/admin/collections?action=CREATE&name=COLL_NAME&collection.configName=COLL_NAME&numShards=1&nrtReplicas=1&wt=javabin&version=2: org.apache.solr.cluster.placement.PlacementException: Not enough eligible nodes to place 1 replica(s) of type NRT for shard shard1 of collection COLL_NAME. Only able to place 0 replicas.
ok 42 Random placement plugin using ENV in 22412ms
ok 43 post solr log into solr in 8886ms
ok 44 start solr with ssl in 19854ms
ok 45 start solr with ssl and auth in 23006ms
ok 46 SOLR-11740 check 'solr stop' connection in 28720ms
ok 47 status detects locally running solr in 14408ms
ok 48 status does not expose cli parameters to end user in 211ms
ok 49 listing out files in 2196ms
ok 50 copying files around in 6657ms
ok 51 upconfig in 2655ms

> Task :solr:packaging:integrationTests FAILED

> Task :solr:core:test

org.apache.solr.TestSolrCoreProperties > classMethod FAILED
    java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([405BE1D6CAE1C16E]:0)

org.apache.solr.TestSolrCoreProperties > test suite's output saved to /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.TestSolrCoreProperties.txt, copied below:
  2> 3889 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.SolrTestCase Setting 'solr.default.confdir' system property to test-framework derived value of '/tmp/src/solr/solr/server/solr/configsets/_default/conf'
  2> 3903 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
  2> 3945 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-1 for ERROR logs matching regex: ignore_exception
  2> 3987 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.SolrTestCaseJ4 Created dataDir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/data-dir-1-001
  2> 4078 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
  2> 4116 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
  2> 4705 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
  2> 4708 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 2 ...
  2> 4718 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.Server jetty-10.0.15; built: 2023-04-11T17:25:14.480Z; git: 68017dbd00236bb7e187330d7585a059610f661d; jvm 11.0.16+8
  2> 4768 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.s.DefaultSessionIdManager Session workerName=node0
  2> 4804 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6e3dfbf4{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,AVAILABLE}
  2> 4828 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.AbstractConnector Started ServerConnector@2a5d9f7e{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39723}
  2> 4832 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.Server Started Server@758bda96{STARTING}[10.0.15,sto=0] @4849ms
  2> 4846 ERROR (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
  2> 4847 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.CoreContainerProvider Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
  2> 4854 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.CoreContainerProvider  ___      _       Welcome to Apache Solr™ version 9.3.0
  2> 4855 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.CoreContainerProvider / __| ___| |_ _   Starting in standalone mode on port null
  2> 4855 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.CoreContainerProvider \__ \/ _ \ | '_|  Install dir: /tmp/src/solr/solr
  2> 4856 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.CoreContainerProvider |___/\___/_|_|    Start time: 2023-06-27T01:36:21.590213Z
  2> 4857 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.CoreContainerProvider Solr Home: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001 (source: servlet config: solr.solr.home)
  2> 4878 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.c.SolrXmlConfig Loading solr.xml from /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001/solr.xml
  2> 10823 WARN  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
  2> 11336 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.c.CorePropertiesLocator Found 1 core definitions underneath /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001
  2> 11338 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.c.CorePropertiesLocator Cores are: [collection1]
  2> 11787 WARN  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig You should not use LATEST as luceneMatchVersion property: if you use this setting, and then Solr upgrades to a newer release of Lucene, sizable changes may happen. If precise back compatibility is important then you should instead explicitly specify an actual Lucene version.
  2> 11806 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.6.0
  2> 11925 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Schema name=tiny
  2> 12224 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.s.IndexSchema Loaded schema tiny/1.1 with uniqueid field id
  2> 12225 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.CoreContainer Creating SolrCore 'collection1' using configuration from instancedir /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001/collection1, trusted=true
  2> 12293 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.c.SolrCore Opening new SolrCore at [/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001/collection1], dataDir=[/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001/collection1/data/]
  2> 12338 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogDocMergePolicy: [LogDocMergePolicy: minMergeSize=1000, mergeFactor=18, maxMergeSize=9223372036854775807, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.0]
  2> 12957 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 13261 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, solr.data.dir=/tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-002, hostPort=39723}
  2> 13483 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.j.SolrRequestAuthorizer Creating a new SolrRequestAuthorizer
  2> 13533 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 13533 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 13539 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.LogByteSizeMergePolicy: [LogByteSizeMergePolicy: minMergeSize=1677721, mergeFactor=11, maxMergeSize=2147483648, maxMergeSizeForForcedMerge=9223372036854775807, calibrateSizeByDeletes=false, maxMergeDocs=2147483647, maxCFSSegmentSizeMB=8.796093022208E12, noCFSRatio=0.0]
  2> 13608 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.r.ManagedResourceStorage File-based storage initialized to use dir: /tmp/src/solr/solr/core/build/tmp/tests-tmp/solr.TestSolrCoreProperties_405BE1D6CAE1C16E-001/tempDir-001/collection1/conf
  2> 13634 INFO  (coreLoadExecutor-11-thread-1) [ x:collection1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 13653 INFO  (searcherExecutor-13-thread-1-processing-collection1) [ x:collection1] o.a.s.c.SolrCore Registered new searcher autowarm time: 0 ms
  2> Jun 27, 2023 10:36:20 AM com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
  2> WARNING: Suite execution timed out: org.apache.solr.TestSolrCoreProperties
  2> ==== jstack at approximately timeout time ====
  2> "Test worker" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@167e00d8
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@167e00d8
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1300)
  2> 	at java.base@11.0.16/java.lang.Thread.join(Thread.java:1375)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:647)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:504)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:57)
  2> 	at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:39)
  2> 	at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
  2> 	at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
  2> 	at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
  2> 	at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
  2> 	at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113)
  2> 	at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
  2> 	at app//worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)
  2> 
  2> "Reference Handler" ID=2 RUNNABLE
  2> 	at java.base@11.0.16/java.lang.ref.Reference.waitForReferencePendingList(Native Method)
  2> 	at java.base@11.0.16/java.lang.ref.Reference.processPendingReferences(Reference.java:241)
  2> 	at java.base@11.0.16/java.lang.ref.Reference$ReferenceHandler.run(Reference.java:213)
  2> 
  2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@5330c05f
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@5330c05f
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:176)
  2> 	at java.base@11.0.16/java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:170)
  2> 
  2> "Signal Dispatcher" ID=4 RUNNABLE
  2> 
  2> "Common-Cleaner" ID=9 TIMED_WAITING on java.lang.ref.ReferenceQueue$Lock@2e61408a
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- timed waiting on java.lang.ref.ReferenceQueue$Lock@2e61408a
  2> 	at java.base@11.0.16/java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:155)
  2> 	at java.base@11.0.16/jdk.internal.ref.CleanerImpl.run(CleanerImpl.java:148)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	at java.base@11.0.16/jdk.internal.misc.InnocuousThread.run(InnocuousThread.java:161)
  2> 
  2> "/127.0.0.1:33246 to /127.0.0.1:46603 workers" ID=11 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2685bf73
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2685bf73
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:367)
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.submitToRun(TestWorker.java:207)
  2> 	- locked org.gradle.api.internal.tasks.testing.worker.TestWorker@260c4f47
  2> 	at org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:169)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
  2> 	at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:182)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:164)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:414)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@1aa7ecca
  2> 
  2> "/127.0.0.1:33246 to /127.0.0.1:46603 workers Thread 2" ID=12 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7362b1c4
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7362b1c4
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at org.gradle.internal.remote.internal.hub.queue.EndPointQueue.take(EndPointQueue.java:49)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionDispatch.run(MessageHub.java:322)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@76505305
  2> 
  2> "/127.0.0.1:33246 to /127.0.0.1:46603 workers Thread 3" ID=13 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@5723da32
  2> 	- locked sun.nio.ch.EPollSelectorImpl@1311c1e6
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection$SocketInputStream.read(SocketConnection.java:185)
  2> 	at com.esotericsoftware.kryo.io.Input.fill(Input.java:146)
  2> 	at com.esotericsoftware.kryo.io.Input.require(Input.java:178)
  2> 	at com.esotericsoftware.kryo.io.Input.readByte(Input.java:295)
  2> 	at org.gradle.internal.serialize.kryo.KryoBackedDecoder.readByte(KryoBackedDecoder.java:88)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:64)
  2> 	at org.gradle.internal.remote.internal.hub.InterHubMessageSerializer$MessageReader.read(InterHubMessageSerializer.java:52)
  2> 	at org.gradle.internal.remote.internal.inet.SocketConnection.receive(SocketConnection.java:81)
  2> 	at org.gradle.internal.remote.internal.hub.MessageHub$ConnectionReceive.run(MessageHub.java:270)
  2> 	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
  2> 	at org.gradle.internal.concurrent.ManagedExecutorImpl$1.run(ManagedExecutorImpl.java:49)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@71d44a3
  2> 
  2> "SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]" ID=14 RUNNABLE
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpThreads0(Native Method)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521)
  2> 	at java.management@11.0.16/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:708)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:59)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:437)
  2> 	- locked java.lang.Object@18648ad0
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:716)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:637)
  2> 
  2> "SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker" ID=15 WAITING on java.util.concurrent.CountDownLatch$Sync@24a1fd3
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.CountDownLatch$Sync@24a1fd3
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2> 	at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2> 	at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2> 	at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2> 	at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2> 	at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2> 	at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner$1@7e6ee489
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2> 	at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2> 	- locked org.apache.solr.embedded.JettySolrRunner@3755533c
  2> 	at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2> 	at app//org.apache.solr.TestSolrCoreProperties.beforeTest(TestSolrCoreProperties.java:80)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2> 	at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2> 	at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2> 	at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2> 	at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2> 	at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2> 	at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2> 	at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2> 	at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2> 	at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2> 	at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021ac40.run(Unknown Source)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@2a56aac1
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@16d88751
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@1f9b63af
  2> 
  2> "qtp970150645-17" ID=17 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@1d55bdf9
  2> 	- locked sun.nio.ch.EPollSelectorImpl@4f65530d
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp970150645-18-acceptor-0@74b5df92-ServerConnector@2a5d9f7e{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39723}" ID=18 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2> 	at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2> 	at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2> 	at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.locks.ReentrantLock$NonfairSync@b27d755
  2> 
  2> "qtp970150645-19" ID=19 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp970150645-20" ID=20 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp970150645-21" ID=21 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp970150645-22" ID=22 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp970150645-23" ID=23 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "qtp970150645-24" ID=24 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@59c1e565
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2> 	at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Session-HouseKeeper-4c57bf2-1" ID=25 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@69cb21d1
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@69cb21d1
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2> 	at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "httpShardExecutor-6-thread-1" ID=26 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@cd3fade
  2> 	- locked sun.nio.ch.EPollSelectorImpl@397b196e
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@5e306632
  2> 
  2> "httpShardExecutor-6-thread-2" ID=27 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@6fc739e9
  2> 	- locked sun.nio.ch.EPollSelectorImpl@5fb25869
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@63944af8
  2> 
  2> "Connection evictor" ID=28 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "Connection evictor" ID=29 TIMED_WAITING
  2> 	at java.base@11.0.16/java.lang.Thread.sleep(Native Method)
  2> 	at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> "h2sc-8-thread-1" ID=30 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@4c244c4b
  2> 	- locked sun.nio.ch.EPollSelectorImpl@2d759cc8
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@5b16524e
  2> 
  2> "h2sc-8-thread-2" ID=31 RUNNABLE (in native code)
  2> 	at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2> 	at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2> 	- locked sun.nio.ch.Util$2@7b2bf0fe
  2> 	- locked sun.nio.ch.EPollSelectorImpl@36178ee2
  2> 	at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2> 	at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2> 	at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2> 	at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 	Locked synchronizers:
  2> 	- java.util.concurrent.ThreadPoolExecutor$Worker@18460e3c
  2> 
  2> "CloserThread" ID=33 WAITING on java.lang.Object@583fca7f
  2> 	at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2> 	- waiting on java.lang.Object@583fca7f
  2> 	at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2> 	at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2> 
  2> "ForkJoinPool.commonPool-worker-3" ID=36 WAITING on java.util.concurrent.ForkJoinPool@59ad9e6c
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.ForkJoinPool@59ad9e6c
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1628)
  2> 	at java.base@11.0.16/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
  2> 
  2> "searcherExecutor-13-thread-1" ID=37 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7da529a7
  2> 	at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7da529a7
  2> 	at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2> 	at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2> 	at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2> 	at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2> 	at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> 
  2> ^^==============================================
  2> 
  2> Jun 27, 2023 10:36:20 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 16 leaked thread(s).
  2> Jun 27, 2023 10:36:21 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> SEVERE: 16 threads leaked from SUITE scope at org.apache.solr.TestSolrCoreProperties: 
  2>    1) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    2) Thread[id=19, name=qtp970150645-19, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    3) Thread[id=24, name=qtp970150645-24, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    4) Thread[id=25, name=Session-HouseKeeper-4c57bf2-1, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
  2>         at java.base@11.0.16/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    5) Thread[id=18, name=qtp970150645-18-acceptor-0@74b5df92-ServerConnector@2a5d9f7e{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39723}, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
  2>         at java.base@11.0.16/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
  2>         at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:409)
  2>         at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:748)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    6) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    7) Thread[id=21, name=qtp970150645-21, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    8) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>    9) Thread[id=15, name=SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker, state=WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
  2>         at java.base@11.0.16/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ContextInitializationKey.waitForReadyService(CoreContainerProvider.java:510)
  2>         at app//org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:549)
  2>         at app//org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147)
  2>         at app//org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472)
  2>         at app//org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992)
  2>         at app//org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480)
  2>         at app//org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253)
  2>         at app//org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554)
  2>         at app//org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525)
  2>         at app//org.apache.solr.TestSolrCoreProperties.beforeTest(TestSolrCoreProperties.java:80)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  2>         at java.base@11.0.16/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  2>         at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  2>         at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886)
  2>         at app//com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  2>         at app//org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  2>         at app//org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  2>         at app//org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  2>         at app//org.junit.rules.RunRules.evaluate(RunRules.java:20)
  2>         at app//com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  2>         at app//com.carrotsearch.randomizedtesting.ThreadLeakControl$$Lambda$186/0x000000084021ac40.run(Unknown Source)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   10) Thread[id=17, name=qtp970150645-17, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   11) Thread[id=22, name=qtp970150645-22, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   12) Thread[id=20, name=qtp970150645-20, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   13) Thread[id=23, name=qtp970150645-23, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
  2>         at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1124)
  2>         at app//org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1141)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   14) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/sun.nio.ch.EPoll.wait(Native Method)
  2>         at java.base@11.0.16/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
  2>         at java.base@11.0.16/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
  2>         at app//org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
  2>         at app//org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
  2>         at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$260/0x0000000840335040.run(Unknown Source)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:289)
  2>         at app//org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$347/0x0000000840416440.run(Unknown Source)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2>   15) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/java.lang.Object.wait(Native Method)
  2>         at java.base@11.0.16/java.lang.Object.wait(Object.java:328)
  2>         at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:2554)
  2>   16) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestSolrCoreProperties]
  2>         at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
  2>         at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
  2>         at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
  2>         at java.base@11.0.16/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
  2>         at java.base@11.0.16/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
  2>         at java.base@11.0.16/java.lang.Thread.run(Thread.java:829)
  2> Jun 27, 2023 10:36:21 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: Starting to interrupt leaked threads:
  2>    1) Thread[id=27, name=httpShardExecutor-6-thread-2, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>    2) Thread[id=19, name=qtp970150645-19, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>    3) Thread[id=24, name=qtp970150645-24, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>    4) Thread[id=25, name=Session-HouseKeeper-4c57bf2-1, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>    5) Thread[id=18, name=qtp970150645-18-acceptor-0@74b5df92-ServerConnector@2a5d9f7e{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:39723}, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>    6) Thread[id=31, name=h2sc-8-thread-2, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>    7) Thread[id=21, name=qtp970150645-21, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>    8) Thread[id=30, name=h2sc-8-thread-1, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>    9) Thread[id=15, name=SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker, state=WAITING, group=TGRP-TestSolrCoreProperties]
  2>   10) Thread[id=17, name=qtp970150645-17, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>   11) Thread[id=22, name=qtp970150645-22, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>   12) Thread[id=20, name=qtp970150645-20, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>   13) Thread[id=23, name=qtp970150645-23, state=TIMED_WAITING, group=TGRP-TestSolrCoreProperties]
  2>   14) Thread[id=26, name=httpShardExecutor-6-thread-1, state=RUNNABLE, group=TGRP-TestSolrCoreProperties]
  2>   15) Thread[id=33, name=CloserThread, state=WAITING, group=TGRP-TestSolrCoreProperties]
  2>   16) Thread[id=37, name=searcherExecutor-13-thread-1, state=WAITING, group=TGRP-TestSolrCoreProperties]
  2> 7204590 ERROR (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.s.SolrDispatchFilter Could not start Dispatch Filter.
  2>           => org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556)
  2> org.apache.solr.common.SolrException: Interrupted while obtaining reference to CoreService
  2> 	at org.apache.solr.servlet.CoreContainerProvider$ServiceHolder.getService(CoreContainerProvider.java:556) ~[main/:?]
  2> 	at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:147) [main/:?]
  2> 	at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:133) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.initializeHolders(ServletHandler.java:774) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.setFilters(ServletHandler.java:1472) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletHandler.addFilterWithMapping(ServletHandler.java:992) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.servlet.ServletContextHandler.addFilter(ServletContextHandler.java:480) [jetty-servlet-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner$1.lifeCycleStarted(JettySolrRunner.java:427) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT ce3b412ef664c5c5837042fc19cf3b8e2aa3240a [snapshot build, details omitted]]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.setStarted(AbstractLifeCycle.java:253) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:94) [jetty-util-10.0.15.jar:10.0.15]
  2> 	at org.apache.solr.embedded.JettySolrRunner.retryOnPortBindFailure(JettySolrRunner.java:616) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT ce3b412ef664c5c5837042fc19cf3b8e2aa3240a [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:554) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT ce3b412ef664c5c5837042fc19cf3b8e2aa3240a [snapshot build, details omitted]]
  2> 	at org.apache.solr.embedded.JettySolrRunner.start(JettySolrRunner.java:525) [solr-test-framework-9.3.0-SNAPSHOT.jar:9.3.0-SNAPSHOT ce3b412ef664c5c5837042fc19cf3b8e2aa3240a [snapshot build, details omitted]]
  2> 	at org.apache.solr.TestSolrCoreProperties.beforeTest(TestSolrCoreProperties.java:80) [test/:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
  2> 	at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
  2> 	at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
  2> 	at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:886) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:80) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47) [lucene-test-framework-9.6.0.jar:9.6.0 f94cd1750d198cd0294fb1d967c4e511a7035f1e - 2023-05-03 10:50:43]
  2> 	at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.13.2.jar:4.13.2]
  2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850) [randomizedtesting-runner-2.8.1.jar:?]
  2> 	at java.lang.Thread.run(Thread.java:829) [?:?]
  2> 7205091 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.Server Stopped Server@758bda96{STOPPING}[10.0.15,sto=0]
  2> 7205095 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.AbstractConnector Stopped ServerConnector@2a5d9f7e{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 7205098 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@6e3dfbf4{/solr,file:///tmp/src/solr/solr/core/build/tmp/tests-cwd/,STOPPED}
  2> 7205104 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.c.CoreContainer Shutting down CoreContainer instance=136008191
  2> 7205107 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.c.SolrCore CLOSING SolrCore org.apache.solr.core.SolrCore@33e0d56b collection1
  2> 7205107 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1 tag=SolrCore@33e0d56b
  2> 7205133 INFO  (coreCloseExecutor-18-thread-1) [ x:collection1] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 7205139 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 7205147 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 7205151 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 7205157 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.u.ErrorLogMuter Closing ErrorLogMuter-regex-1 after mutting 0 log messages
  2> 7205158 INFO  (SUITE-TestSolrCoreProperties-seed#[405BE1D6CAE1C16E]-worker) [] o.a.s.u.ErrorLogMuter Creating ErrorLogMuter-regex-2 for ERROR logs matching regex: ignore_exception
  2> Jun 27, 2023 3:36:22 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
  2> INFO: All leaked threads terminated.
   >     java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   >         at __randomizedtesting.SeedInfo.seed([405BE1D6CAE1C16E]:0)
  2> NOTE: test params are: codec=Asserting(Lucene95): {}, docValues:{}, maxPointsInLeafNode=354, maxMBSortInHeap=6.294841623997741, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=prg, timezone=Asia/Krasnoyarsk
  2> NOTE: Linux 5.4.0-100-generic amd64/Oracle Corporation 11.0.16 (64-bit)/cpus=1,threads=1,free=279877456,total=342360064
  2> NOTE: All tests run in this JVM: [TestSolrCoreProperties]
  2> NOTE: reproduce with: gradlew test --tests TestSolrCoreProperties -Dtests.seed=405BE1D6CAE1C16E -Dtests.locale=prg -Dtests.timezone=Asia/Krasnoyarsk -Dtests.asserts=true -Dtests.file.encoding=UTF-8

:solr:core:test (FAILURE): 4796 test(s), 1 failure(s), 203 skipped

4796 tests completed, 1 failed, 203 skipped

> Task :solr:core:test FAILED
> Task :solr:core:wipeTaskTemp

ERROR: The following test(s) have failed:
  - org.apache.solr.TestSolrCoreProperties.classMethod (:solr:core)
    Test output: /tmp/src/solr/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.TestSolrCoreProperties.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.TestSolrCoreProperties.classMethod" -Ptests.jvms=96 "-Ptests.jvmargs=-XX:TieredStopAtLevel=1 -XX:+UseParallelGC -XX:ActiveProcessorCount=1 -XX:ReservedCodeCacheSize=120m" -Ptests.seed=405BE1D6CAE1C16E -Ptests.file.encoding=US-ASCII


FAILURE: Build completed with 2 failures.

1: Task failed with an exception.
-----------
* Where:
Build file '/tmp/src/solr/solr/packaging/build.gradle' line: 285

* What went wrong:
Execution failed for task ':solr:packaging:integrationTests'.
> Process 'command '/tmp/src/solr/.gradle/node/packaging/node_modules/bats/bin/bats'' finished with non-zero exit value 1

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

2: Task failed with an exception.
-----------
* What went wrong:
Execution failed for task ':solr:core:test'.
> There were failing tests. See the results at: file:///tmp/src/solr/solr/core/build/test-results/test/

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
==============================================================================

* Get more help at https://help.gradle.org

BUILD FAILED in 2h 4m 42s
625 actionable tasks: 609 executed, 16 up-to-date
[?12l[?25h
------------------------------------------------------------------------
Build Failed: returned 1

Total time: 2h5m9s
------------------------------------------------------------------------
+ status=1
+ ./crave pull '**/build/**/test/TEST-*.xml' '**/*.events' 'heapdumps/**' '**/hs_err_pid*'
Error: rsync: [sender] change_dir "/tmp/src/solr/heapdumps" failed: No such file or directory (2)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1675) [Receiver=3.1.2]
rsync: [Receiver] write error: Broken pipe (32)

+ exit 1
Build step 'Execute shell' marked build as failure
Archiving artifacts
Recording test results
[Checks API] No suitable checks publisher found.
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any