You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2020/09/23 21:22:01 UTC

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-12.0.2) - Build # 6564 - Still Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/6564/
Java: 64bit/jdk-12.0.2 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestQueryingOnDownCollection.classMethod

Error Message:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestQueryingOnDownCollection: 
   1) Thread[id=23271, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestQueryingOnDownCollection]
        at java.base@12.0.2/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:157)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.TestQueryingOnDownCollection: 
   1) Thread[id=23271, name=SessionTracker, state=TIMED_WAITING, group=TGRP-TestQueryingOnDownCollection]
        at java.base@12.0.2/java.lang.Thread.sleep(Native Method)
        at app//org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:157)
	at __randomizedtesting.SeedInfo.seed([ED0912041BA28FF4]:0)




Build Log:
[...truncated 2001 lines...]
ERROR: The following test(s) have failed:
  - org.apache.solr.cloud.TestQueryingOnDownCollection.classMethod (:solr:core)
    Test output: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.cloud.TestQueryingOnDownCollection.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.cloud.TestQueryingOnDownCollection" -Ptests.jvms=5 -Ptests.haltonfailure=false -Ptests.jvmargs=-XX:+UseCompressedOops -XX:+UseSerialGC -Ptests.seed=ED0912041BA28FF4 -Ptests.badapples=false -Ptests.file.encoding=US-ASCII


Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.4.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1h 14m 25s
837 actionable tasks: 837 executed
Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2875)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2754)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2735)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3122)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to MacOSX VBOX
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at hudson.FilePath.act(FilePath.java:1070)
		at hudson.FilePath.act(FilePath.java:1059)
		at hudson.FilePath.validateAntFileMask(FilePath.java:2733)
		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
		at hudson.model.Build$BuildExecution.post2(Build.java:177)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
		at hudson.model.Run.execute(Run.java:1919)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:428)
Caused: hudson.FilePath$TunneledInterruptedException
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3124)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:375)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.validateAntFileMask(FilePath.java:2733)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
	at hudson.model.Build$BuildExecution.post2(Build.java:177)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
	at hudson.model.Run.execute(Run.java:1919)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:428)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[Java] Sleeping for 5 seconds due to JENKINS-32191...
[Java] Parsing console log (workspace: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX')
[Java] Successfully parsed console log
[Java] -> found 0 issues (skipped 0 duplicates)
[Java] Skipping post processing
[Java] No filter has been set, publishing all 0 issues
[Java] Using reference build 'Lucene-Solr-master-MacOSX #6563' to compute new, fixed, and outstanding issues
[Java] Issues delta (vs. reference build): outstanding: 0, new: 0, fixed: 0
[Java] No quality gates have been set - skipping
[Java] Health report is disabled - skipping
[Java] Created analysis result for 0 issues (found 0 new issues, fixed 0 issues)
[Java] Attaching ResultAction with ID 'java' to run 'Lucene-Solr-master-MacOSX #6564'.
[Checks API] No suitable checks publisher found.
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
[Checks API] No suitable checks publisher found.
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-15) - Build # 6566 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/6566/
Java: 64bit/jdk-15 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.handler.component.CustomHighlightComponentTest.test

Error Message:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:54198/solr/collection76: error processing commands

Stack Trace:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:54198/solr/collection76: error processing commands
	at __randomizedtesting.SeedInfo.seed([ED53A5875DD0FD37:65079A5DF32C90CF]:0)
	at org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException.create(BaseHttpSolrClient.java:67)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:664)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
	at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
	at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
	at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1170)
	at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
	at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
	at org.apache.solr.handler.component.CustomHighlightComponentTest.test(CustomHighlightComponentTest.java:163)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
	at java.base/java.lang.Thread.run(Thread.java:832)




Build Log:
[...truncated 8532 lines...]
  2> 177919 ERROR (qtp503089417-1028) [n:127.0.0.1:54198_solr c:collection76 s:shard2 r:core_node7 x:collection76_shard2_replica_n4 ] o.a.s.h.SolrConfigHandler ERROR:[{
  2>     "add-requesthandler":{
  2>       "name":"/custom_select-1536458086",
  2>       "class":"org.apache.solr.handler.component.SearchHandler",
  2>       "components":[
  2>         "query",
  2>         "highlight"]},
  2>     "errorMessages":[" '/custom_select-1536458086' already exists . Do an 'update-requesthandler' , if you want to change it "]}]
  2> 177920 INFO  (qtp503089417-1028) [n:127.0.0.1:54198_solr c:collection76 s:shard2 r:core_node7 x:collection76_shard2_replica_n4 ] o.a.s.c.S.Request [collection76_shard2_replica_n4]  webapp=/solr path=/config params={wt=javabin&version=2} status=400 QTime=16
  2> 177922 ERROR (TEST-CustomHighlightComponentTest.test-seed#[ED53A5875DD0FD37]) [     ] o.a.s.c.s.i.BaseCloudSolrClient Request to collection [collection76] failed due to (400) org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:54198/solr/collection76: error processing commands, retry=0 commError=false errorCode=400 
  2> 177924 INFO  (TEST-CustomHighlightComponentTest.test-seed#[ED53A5875DD0FD37]) [     ] o.a.s.c.s.i.BaseCloudSolrClient request was not communication error it seems
  2> 177927 INFO  (TEST-CustomHighlightComponentTest.test-seed#[ED53A5875DD0FD37]) [     ] o.a.s.SolrTestCaseJ4 ###Ending test
   >     org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:54198/solr/collection76: error processing commands
   >         at __randomizedtesting.SeedInfo.seed([ED53A5875DD0FD37:65079A5DF32C90CF]:0)
   >         at org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException.create(BaseHttpSolrClient.java:67)
   >         at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:664)
   >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
   >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
   >         at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
   >         at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
   >         at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1170)
   >         at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
   >         at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
   >         at org.apache.solr.handler.component.CustomHighlightComponentTest.test(CustomHighlightComponentTest.java:163)
   >         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
   >         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >         at java.base/java.lang.reflect.Method.invoke(Method.java:564)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
   >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   >         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   >         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   >         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   >         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   >         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   >         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   >         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   >         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   >         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
   >         at java.base/java.lang.Thread.run(Thread.java:832)
  2> NOTE: reproduce with: gradlew test --tests CustomHighlightComponentTest.test -Dtests.seed=ED53A5875DD0FD37 -Dtests.slow=true -Dtests.locale=uz-Latn-UZ -Dtests.timezone=Pacific/Galapagos -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> 178338 INFO  (searcherExecutor-800-thread-1-processing-n:127.0.0.1:54197_solr) [n:127.0.0.1:54197_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n1]  Registered new searcher autowarm time: 0 ms
  2> 178340 INFO  (qtp338216678-1031) [n:127.0.0.1:54197_solr c:collection76 s:shard1 r:core_node3 x:collection76_shard1_replica_n1 ] o.a.s.h.SolrConfigHandler I already have the expected version 0 of config
  2> 178340 INFO  (qtp338216678-1031) [n:127.0.0.1:54197_solr c:collection76 s:shard1 r:core_node3 x:collection76_shard1_replica_n1 ] o.a.s.h.SolrConfigHandler isStale false , resourceloader org.apache.solr.cloud.ZkSolrResourceLoader
  2> 178340 INFO  (qtp338216678-1031) [n:127.0.0.1:54197_solr c:collection76 s:shard1 r:core_node3 x:collection76_shard1_replica_n1 ] o.a.s.c.S.Request [collection76_shard1_replica_n1]  webapp=/solr path=/config/znodeVersion params={overlay=0&wt=javabin&version=2} status=0 QTime=1190
  2> 178488 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=327490685
  2> 178488 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54200_solr
  2> 178490 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=444877491
  2> 178491 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54202_solr
  2> 178493 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2074611475
  2> 178494 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54199_solr
  2> 178496 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=719294350
  2> 178497 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54198_solr
  2> 178508 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1119315164
  2> 178509 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54197_solr
  2> 178638 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 178638 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54198_solr as DOWN
  2> 178639 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 178639 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54199_solr as DOWN
  2> 178643 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 178643 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54200_solr as DOWN
  2> 178645 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 178645 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54197_solr as DOWN
  2> 178647 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 178647 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54202_solr as DOWN
  2> 178859 INFO  (zkCallback-651-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (1)
  2> 178859 INFO  (zkCallback-657-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (1)
  2> 178873 INFO  (zkCallback-630-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (1)
  2> 178874 INFO  (zkCallback-660-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (1)
  2> 178875 INFO  (zkCallback-627-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (1)
  2> 178886 INFO  (zkCallback-670-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (1)
  2> 179088 INFO  (coreCloseExecutor-2088-thread-1) [n:127.0.0.1:54202_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n10]  CLOSING SolrCore org.apache.solr.core.SolrCore@6231a0cd
  2> 179088 INFO  (coreCloseExecutor-2088-thread-1) [n:127.0.0.1:54202_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n10 tag=SolrCore@6231a0cd
  2> 179088 INFO  (coreCloseExecutor-2088-thread-1) [n:127.0.0.1:54202_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@190b7d4: rootName = solr_54202, domain = solr.core.collection76.shard3.replica_n10, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n10/com.codahale.metrics.MetricRegistry@20cc59cc
  2> 179095 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=219382452
  2> 179095 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54201_solr
  2> 179138 INFO  (coreCloseExecutor-2093-thread-1) [n:127.0.0.1:54199_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n6]  CLOSING SolrCore org.apache.solr.core.SolrCore@393d1d0c
  2> 179139 INFO  (coreCloseExecutor-2093-thread-1) [n:127.0.0.1:54199_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n6 tag=SolrCore@393d1d0c
  2> 179139 INFO  (coreCloseExecutor-2093-thread-1) [n:127.0.0.1:54199_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@18cbcd2a: rootName = solr_54199, domain = solr.core.collection76.shard2.replica_n6, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n6/com.codahale.metrics.MetricRegistry@614738cc
  2> 179139 INFO  (coreCloseExecutor-2094-thread-1) [n:127.0.0.1:54197_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@252b280
  2> 179140 INFO  (coreCloseExecutor-2095-thread-1) [n:127.0.0.1:54200_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n8]  CLOSING SolrCore org.apache.solr.core.SolrCore@6daf2c3a
  2> 179140 INFO  (coreCloseExecutor-2095-thread-1) [n:127.0.0.1:54200_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n8 tag=SolrCore@6daf2c3a
  2> 179140 INFO  (coreCloseExecutor-2095-thread-1) [n:127.0.0.1:54200_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2d3145e2: rootName = solr_54200, domain = solr.core.collection76.shard3.replica_n8, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n8/com.codahale.metrics.MetricRegistry@694dbdf3
  2> 179160 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 179161 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:54201_solr as DOWN
  2> 179198 INFO  (coreCloseExecutor-2096-thread-1) [n:127.0.0.1:54198_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  CLOSING SolrCore org.apache.solr.core.SolrCore@5934f850
  2> 179198 INFO  (coreCloseExecutor-2096-thread-1) [n:127.0.0.1:54198_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n4 tag=SolrCore@5934f850
  2> 179198 INFO  (coreCloseExecutor-2096-thread-1) [n:127.0.0.1:54198_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5057d012: rootName = solr_54198, domain = solr.core.collection76.shard2.replica_n4, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n4/com.codahale.metrics.MetricRegistry@76d22d27
  2> 179200 INFO  (zkCallback-660-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 179203 INFO  (zkCallback-670-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 179210 INFO  (zkCallback-651-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 179231 INFO  (zkCallback-627-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 179231 INFO  (zkCallback-657-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 179231 INFO  (zkCallback-630-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 179270 INFO  (searcherExecutor-802-thread-1-processing-n:127.0.0.1:54200_solr) [n:127.0.0.1:54200_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n8]  Registered new searcher autowarm time: 0 ms
  2> 179276 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54200_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n8]  CLOSING SolrCore org.apache.solr.core.SolrCore@3ac97f44
  2> 179333 INFO  (coreCloseExecutor-2098-thread-1) [n:127.0.0.1:54201_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@17a006a6
  2> 179333 INFO  (coreCloseExecutor-2098-thread-1) [n:127.0.0.1:54201_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n2 tag=SolrCore@17a006a6
  2> 179334 INFO  (coreCloseExecutor-2098-thread-1) [n:127.0.0.1:54201_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1ba62eda: rootName = solr_54201, domain = solr.core.collection76.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n2/com.codahale.metrics.MetricRegistry@59283bea
  2> 179427 INFO  (zkCallback-660-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179427 INFO  (zkCallback-651-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179427 INFO  (zkCallback-630-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179428 INFO  (zkCallback-630-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179428 INFO  (zkCallback-660-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179427 INFO  (zkCallback-670-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179430 INFO  (zkCallback-651-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179433 INFO  (zkCallback-657-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179433 INFO  (zkCallback-657-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179442 INFO  (zkCallback-627-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179716 INFO  (zkCallback-651-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179717 INFO  (zkCallback-630-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179716 INFO  (zkCallback-630-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179716 INFO  (zkCallback-651-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179724 INFO  (zkCallback-670-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179724 INFO  (zkCallback-627-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179724 INFO  (zkCallback-660-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179725 INFO  (zkCallback-660-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179740 INFO  (zkCallback-657-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 179741 INFO  (zkCallback-657-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 180341 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 180341 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 180401 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 180404 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 180405 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 180408 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678702441496838144
  2> 180423 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 180487 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 180487 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 180991 INFO  (searcherExecutor-802-thread-1-processing-n:127.0.0.1:54200_solr) [n:127.0.0.1:54200_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n8]  Registered new searcher autowarm time: 0 ms
  2> 181345 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 181345 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 181384 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 181389 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 181393 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 181394 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678702442530734080
  2> 181396 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 181568 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 181574 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 181576 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 181576 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678702442721574912
  2> 181622 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 181823 INFO  (searcherExecutor-804-thread-1-processing-n:127.0.0.1:54201_solr) [n:127.0.0.1:54201_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n2]  Registered new searcher autowarm time: 0 ms
  2> 181825 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n2]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c59e95f
  2> 181954 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 181960 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 181961 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 181961 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678702443125276672
  2> 181976 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 182034 INFO  (searcherExecutor-814-thread-1-processing-n:127.0.0.1:54202_solr) [n:127.0.0.1:54202_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n10]  Registered new searcher autowarm time: 0 ms
  2> 182036 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n10]  CLOSING SolrCore org.apache.solr.core.SolrCore@5e21870c
  2> 182313 INFO  (searcherExecutor-810-thread-1-processing-n:127.0.0.1:54198_solr) [n:127.0.0.1:54198_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  Registered new searcher autowarm time: 0 ms
  2> 182319 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  CLOSING SolrCore org.apache.solr.core.SolrCore@54fa593c
  2> 182350 INFO  (searcherExecutor-804-thread-1-processing-n:127.0.0.1:54201_solr) [n:127.0.0.1:54201_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n2]  Registered new searcher autowarm time: 0 ms
  2> 182394 INFO  (searcherExecutor-808-thread-1-processing-n:127.0.0.1:54199_solr) [n:127.0.0.1:54199_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n6]  Registered new searcher autowarm time: 0 ms
  2> 182405 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n6]  CLOSING SolrCore org.apache.solr.core.SolrCore@12ad591b
  2> 182431 INFO  (searcherExecutor-814-thread-1-processing-n:127.0.0.1:54202_solr) [n:127.0.0.1:54202_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n10]  Registered new searcher autowarm time: 0 ms
  2> 182616 INFO  (searcherExecutor-810-thread-1-processing-n:127.0.0.1:54198_solr) [n:127.0.0.1:54198_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  Registered new searcher autowarm time: 0 ms
  2> 182857 INFO  (searcherExecutor-808-thread-1-processing-n:127.0.0.1:54199_solr) [n:127.0.0.1:54199_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n6]  Registered new searcher autowarm time: 0 ms
  2> 184575 INFO  (Thread-320) [n:127.0.0.1:54197_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@4041e3ba
  2> 184575 INFO  (coreCloseExecutor-2094-thread-1) [n:127.0.0.1:54197_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n1 tag=SolrCore@252b280
  2> 184576 INFO  (coreCloseExecutor-2094-thread-1) [n:127.0.0.1:54197_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@76ddccdb: rootName = solr_54197, domain = solr.core.collection76.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n1/com.codahale.metrics.MetricRegistry@691f482d
  2> 184934 INFO  (coreCloseExecutor-2088-thread-1) [n:127.0.0.1:54202_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@6231a0cd
  2> 184934 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n10 tag=SolrCore@5e21870c
  2> 184934 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6cc59b30: rootName = solr_54202, domain = solr.core.collection76.shard3.replica_n10, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n10/com.codahale.metrics.MetricRegistry@20cc59cc
  2> 184999 INFO  (coreCloseExecutor-2096-thread-1) [n:127.0.0.1:54198_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@5934f850
  2> 184999 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n4 tag=SolrCore@54fa593c
  2> 184999 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@222de840: rootName = solr_54198, domain = solr.core.collection76.shard2.replica_n4, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n4/com.codahale.metrics.MetricRegistry@76d22d27
  2> 185015 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n6 tag=SolrCore@12ad591b
  2> 185015 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1a941030: rootName = solr_54199, domain = solr.core.collection76.shard2.replica_n6, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n6/com.codahale.metrics.MetricRegistry@614738cc
  2> 185056 INFO  (coreCloseExecutor-2095-thread-1) [n:127.0.0.1:54200_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@6daf2c3a
  2> 185056 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54200_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n8 tag=SolrCore@3ac97f44
  2> 185056 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54200_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3b87769b: rootName = solr_54200, domain = solr.core.collection76.shard3.replica_n8, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n8/com.codahale.metrics.MetricRegistry@694dbdf3
  2> 185113 INFO  (coreCloseExecutor-2098-thread-1) [n:127.0.0.1:54201_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@17a006a6
  2> 185113 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n2 tag=SolrCore@1c59e95f
  2> 185114 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3fb3b851: rootName = solr_54201, domain = solr.core.collection76.shard1.replica_n2, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n2/com.codahale.metrics.MetricRegistry@59283bea
  2> 191831 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54200_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@3ac97f44
  2> 191831 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 191831 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@589a288d: rootName = solr_54200, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@37be747f
  2> 191832 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54200_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 191836 ERROR (SolrConfigHandler-refreshconf) [n:127.0.0.1:54200_solr     ] o.a.s.h.SolrConfigHandler Unable to refresh conf 
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard3_replica_n8]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard3_replica_n8]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1596) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.handler.SolrConfigHandler$Command.lambda$handleGET$0(SolrConfigHandler.java:234) [main/:?]
  2> 	at java.lang.Thread.run(Thread.java:832) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1175) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1638) ~[main/:?]
  2> 	... 4 more
  2> 192120 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@12ad591b
  2> 192120 INFO  (coreCloseExecutor-2093-thread-1) [n:127.0.0.1:54199_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@393d1d0c
  2> 192122 ERROR (SolrConfigHandler-refreshconf) [n:127.0.0.1:54199_solr     ] o.a.s.h.SolrConfigHandler Unable to refresh conf 
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n6]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n6]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1596) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.handler.SolrConfigHandler$Command.lambda$handleGET$0(SolrConfigHandler.java:234) [main/:?]
  2> 	at java.lang.Thread.run(Thread.java:832) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1175) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1638) ~[main/:?]
  2> 	... 4 more
  2> 192124 INFO  (coreCloseExecutor-2093-thread-1) [n:127.0.0.1:54199_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 192128 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 192128 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@95a84f4: rootName = solr_54199, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@6561ebde
  2> 192138 INFO  (coreCloseExecutor-2094-thread-1) [n:127.0.0.1:54197_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@252b280
  2> 192139 INFO  (coreCloseExecutor-2094-thread-1) [n:127.0.0.1:54197_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 192144 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 192144 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@75ba15d5: rootName = solr_54197, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@a65d96d
  2> 192240 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 192240 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5a9aea5d: rootName = solr_54202, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@183f7104
  2> 192374 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 192374 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6b59bc6b: rootName = solr_54198, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@30ce5d05
  2> 192416 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 192416 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7b8f143: rootName = solr_54201, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@7335a8bd
  2> 193252 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 193252 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@463867fb: rootName = solr_54200, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@51be65c4
  2> 193566 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 193566 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@725b85a7: rootName = solr_54199, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@51be65c4
  2> 193578 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 193578 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b51610c: rootName = solr_54197, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@51be65c4
  2> 193615 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 193615 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@10f5758b: rootName = solr_54202, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@51be65c4
  2> 193643 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 193644 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@d564585: rootName = solr_54200, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@567de44b
  2> 193644 INFO  (jetty-closer-2074-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 193664 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 193664 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6535fcb5: rootName = solr_54198, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@51be65c4
  2> 193677 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 193678 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@da8aa8a: rootName = solr_54201, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@51be65c4
  2> 193774 INFO  (jetty-closer-2074-thread-3) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@6228a7cc{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 193778 INFO  (jetty-closer-2074-thread-3) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1062edd5{/solr,null,UNAVAILABLE}
  2> 193779 INFO  (jetty-closer-2074-thread-3) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 193880 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 193880 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5f6acf29: rootName = solr_54199, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@567de44b
  2> 193880 INFO  (jetty-closer-2074-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 193888 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 193889 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@53bef461: rootName = solr_54197, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@567de44b
  2> 193889 INFO  (jetty-closer-2074-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 193906 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@5e21870c
  2> 193906 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 193906 INFO  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 193906 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@69556de9: rootName = solr_54202, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@567de44b
  2> 193907 INFO  (jetty-closer-2074-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 193910 WARN  (Thread-321) [n:127.0.0.1:54202_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard3_replica_n10]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard3_replica_n10]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1596) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT cafa449769fb131830ede129910287670625aa0d [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:832) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1175) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1638) ~[main/:?]
  2> 	... 4 more
  2> 193919 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@54fa593c
  2> 193920 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 193920 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@583a73c3: rootName = solr_54201, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@567de44b
  2> 193920 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 193920 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@1c59e95f
  2> 193920 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2a5fa790: rootName = solr_54198, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@567de44b
  2> 193920 INFO  (jetty-closer-2074-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 193920 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 193920 INFO  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 193920 INFO  (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 193925 INFO  (closeThreadPool-2115-thread-1) [     ] o.a.s.c.Overseer Overseer (id=72074367453495307-127.0.0.1:54201_solr-n_0000000000) closing
  2> 193926 INFO  (OverseerStateUpdate-72074367453495307-127.0.0.1:54201_solr-n_0000000000) [n:127.0.0.1:54201_solr     ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:54201_solr
  2> 193927 WARN  (Thread-319) [n:127.0.0.1:54201_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard1_replica_n2]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard1_replica_n2]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1596) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT cafa449769fb131830ede129910287670625aa0d [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:832) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1175) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1638) ~[main/:?]
  2> 	... 4 more
  2> 193929 ERROR (SolrConfigHandler-refreshconf) [n:127.0.0.1:54198_solr     ] o.a.s.h.SolrConfigHandler Unable to refresh conf 
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n4]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n4]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1677) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1596) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.handler.SolrConfigHandler$Command.lambda$handleGET$0(SolrConfigHandler.java:234) [main/:?]
  2> 	at java.lang.Thread.run(Thread.java:832) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1175) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1638) ~[main/:?]
  2> 	... 4 more
  2> 193930 INFO  (closeThreadPool-2115-thread-2) [     ] o.a.s.c.Overseer Overseer (id=72074367453495307-127.0.0.1:54201_solr-n_0000000000) closing
  2> 193935 WARN  (jetty-launcher-599-thread-6-SendThread(127.0.0.1:54191)) [     ] o.a.z.ClientCnxn An exception was thrown while closing send thread for session 0x1000f415d7a000b.
  2>           => EndOfStreamException: Unable to read additional data from server sessionid 0x1000f415d7a000b, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
  2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x1000f415d7a000b, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275) [zookeeper-3.6.2.jar:3.6.2]
  2> 193999 INFO  (jetty-closer-2074-thread-5) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@7c545892{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 193999 INFO  (jetty-closer-2074-thread-5) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1adaaaab{/solr,null,UNAVAILABLE}
  2> 193999 INFO  (jetty-closer-2074-thread-5) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 194092 INFO  (jetty-closer-2074-thread-1) [     ] o.a.s.c.Overseer Overseer (id=72074367453495307-127.0.0.1:54201_solr-n_0000000000) closing
  2> 194125 INFO  (jetty-closer-2074-thread-1) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@60925f1f{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 194125 INFO  (jetty-closer-2074-thread-1) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@36692736{/solr,null,UNAVAILABLE}
  2> 194125 INFO  (jetty-closer-2074-thread-1) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 194210 INFO  (jetty-closer-2074-thread-6) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@133b64c6{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 194211 INFO  (jetty-closer-2074-thread-6) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5b92a734{/solr,null,UNAVAILABLE}
  2> 194211 INFO  (jetty-closer-2074-thread-6) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 194341 INFO  (jetty-closer-2074-thread-4) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@5d8e14b{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 194341 INFO  (jetty-closer-2074-thread-4) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4ff74731{/solr,null,UNAVAILABLE}
  2> 194341 INFO  (jetty-closer-2074-thread-4) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 194420 INFO  (jetty-closer-2074-thread-2) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@1aef2a13{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:0}
  2> 194421 INFO  (jetty-closer-2074-thread-2) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@19665ac2{/solr,null,UNAVAILABLE}
  2> 194421 INFO  (jetty-closer-2074-thread-2) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 194423 INFO  (SUITE-CustomHighlightComponentTest-seed#[ED53A5875DD0FD37]-worker) [     ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
  2> 194538 WARN  (SUITE-CustomHighlightComponentTest-seed#[ED53A5875DD0FD37]-worker-SendThread(127.0.0.1:54191)) [     ] o.a.z.ClientCnxn An exception was thrown while closing send thread for session 0x1000f415d7a0001.
  2>           => EndOfStreamException: Unable to read additional data from server sessionid 0x1000f415d7a0001, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
  2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x1000f415d7a0001, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275) [zookeeper-3.6.2.jar:3.6.2]
  2> 194649 WARN  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Watch limit violations: 
  2> Maximum concurrent create/delete watches above limit:
  2> 
  2> 	12	/solr/configs/conf
  2> 	7	/solr/aliases.json
  2> 	7	/solr/clusterprops.json
  2> 	6	/solr/collections/collection76/terms/shard2
  2> 	6	/solr/packages.json
  2> 	6	/solr/security.json
  2> 	5	/solr/collections/collection76/terms/shard1
  2> 	5	/solr/collections/collection76/terms/shard3
  2> 
  2> Maximum concurrent data watches above limit:
  2> 
  2> 	71	/solr/collections/collection76/state.json
  2> 	2	/solr/overseer_elect/election/72074367453495310-127.0.0.1:54198_solr-n_0000000002
  2> 	2	/solr/overseer_elect/election/72074367453495307-127.0.0.1:54201_solr-n_0000000000
  2> 
  2> Maximum concurrent children watches above limit:
  2> 
  2> 	42	/solr/live_nodes
  2> 	32	/solr/overseer/queue
  2> 	14	/solr/collections
  2> 	3	/solr/overseer/collection-queue-work
  2> 
  2> 194666 INFO  (SUITE-CustomHighlightComponentTest-seed#[ED53A5875DD0FD37]-worker) [     ] o.a.s.c.ZkTestServer waitForServerDown: 127.0.0.1:54191
  2> 194666 INFO  (SUITE-CustomHighlightComponentTest-seed#[ED53A5875DD0FD37]-worker) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:54191
  2> 194666 INFO  (SUITE-CustomHighlightComponentTest-seed#[ED53A5875DD0FD37]-worker) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 54191
  2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/tmp/tests-tmp/solr.handler.component.CustomHighlightComponentTest_ED53A5875DD0FD37-001
  2> Sep 24, 2020 8:16:57 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
  2> NOTE: test params are: codec=Asserting(Lucene87): {}, docValues:{}, maxPointsInLeafNode=1739, maxMBSortInHeap=7.976145478675121, sim=Asserting(RandomSimilarity(queryNorm=false): {}), locale=uz-Latn-UZ, timezone=Pacific/Galapagos
  2> NOTE: Mac OS X 10.14.6 x86_64/Oracle Corporation 15 (64-bit)/cpus=6,threads=1,free=48319608,total=259784704
  2> NOTE: All tests run in this JVM: [V2StandaloneTest, XsltUpdateRequestHandlerTest, TestHdfsBackupRestoreCore, CoreAdminRequestStatusTest, ShowFileRequestHandlerTest, DaemonStreamApiTest, ZookeeperReadAPITest, IndexSizeEstimatorTest, TestCollectionAPIs, TestPivotHelperCode, SpellCheckComponentTest, QueryElevationComponentTest, CustomHighlightComponentTest]

WARNING: Test org.apache.solr.TestDistributedSearch wrote 28,170,482 bytes of output.
WARNING: Test org.apache.solr.search.facet.TestCloudJSONFacetSKGEquiv wrote 17,342,969 bytes of output.
WARNING: Test org.apache.solr.search.facet.TestCloudJSONFacetSKG wrote 25,110,871 bytes of output.
WARNING: Test org.apache.solr.search.facet.TestJsonFacets wrote 28,593,869 bytes of output.
WARNING: Test org.apache.solr.schema.TestCloudSchemaless wrote 23,922,647 bytes of output.
:solr:core:test (FAILURE): 4547 test(s), 1 failure(s), 363 skipped

4547 tests completed, 1 failed, 363 skipped
There were failing tests. See the results at: file:///Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/test-results/test/

> Task :solr:server:test NO-SOURCE
> Task :solr:solr-ref-guide:test NO-SOURCE
> Task :solr:solrj:test
:solr:solrj:test (SUCCESS): 1665 test(s), 21 skipped

> Task :solr:test-framework:test
:solr:test-framework:test (SUCCESS): 9 test(s)

> Task :solr:webapp:test NO-SOURCE

> Task :lucene:analysis:common:test
:lucene:analysis:common:test (SUCCESS): 1639 test(s), 9 skipped

> Task :lucene:analysis:icu:test
:lucene:analysis:icu:test (SUCCESS): 149 test(s), 11 skipped

> Task :lucene:analysis:kuromoji:test
:lucene:analysis:kuromoji:test (SUCCESS): 159 test(s), 7 skipped

> Task :lucene:analysis:morfologik:test
:lucene:analysis:morfologik:test (SUCCESS): 18 test(s)

> Task :lucene:analysis:nori:test
:lucene:analysis:nori:test (SUCCESS): 69 test(s), 4 skipped

> Task :lucene:analysis:opennlp:test
:lucene:analysis:opennlp:test (SUCCESS): 26 test(s)

> Task :lucene:analysis:phonetic:test
:lucene:analysis:phonetic:test (SUCCESS): 41 test(s)

> Task :lucene:analysis:smartcn:test
:lucene:analysis:smartcn:test (SUCCESS): 21 test(s)

> Task :lucene:analysis:stempel:test
:lucene:analysis:stempel:test (SUCCESS): 15 test(s)

> Task :solr:contrib:analysis-extras:test
:solr:contrib:analysis-extras:test (SUCCESS): 30 test(s)

> Task :solr:contrib:analytics:test
:solr:contrib:analytics:test (SUCCESS): 472 test(s)

> Task :solr:contrib:clustering:test
:solr:contrib:clustering:test (SUCCESS): 30 test(s)

> Task :solr:contrib:extraction:test
:solr:contrib:extraction:test (SUCCESS): 20 test(s)

> Task :solr:contrib:jaegertracer-configurator:test
:solr:contrib:jaegertracer-configurator:test (SUCCESS): 3 test(s)

> Task :solr:contrib:langid:test
:solr:contrib:langid:test (SUCCESS): 43 test(s)

> Task :solr:contrib:ltr:test
:solr:contrib:ltr:test (SUCCESS): 127 test(s), 3 skipped

> Task :solr:contrib:prometheus-exporter:test
:solr:contrib:prometheus-exporter:test (SUCCESS): 17 test(s), 5 skipped

> Task :solr:contrib:prometheus-exporter:cleanTaskTmp
> Task :solr:contrib:ltr:cleanTaskTmp
> Task :solr:contrib:langid:cleanTaskTmp
> Task :solr:contrib:jaegertracer-configurator:cleanTaskTmp
> Task :solr:contrib:extraction:cleanTaskTmp
> Task :solr:contrib:clustering:cleanTaskTmp
> Task :solr:contrib:analytics:cleanTaskTmp
> Task :solr:contrib:analysis-extras:cleanTaskTmp
> Task :lucene:analysis:stempel:cleanTaskTmp
> Task :lucene:analysis:smartcn:cleanTaskTmp
> Task :lucene:analysis:phonetic:cleanTaskTmp
> Task :lucene:analysis:opennlp:cleanTaskTmp
> Task :lucene:analysis:nori:cleanTaskTmp
> Task :lucene:analysis:morfologik:cleanTaskTmp
> Task :lucene:analysis:kuromoji:cleanTaskTmp
> Task :lucene:analysis:icu:cleanTaskTmp
> Task :lucene:analysis:common:cleanTaskTmp
> Task :solr:webapp:cleanTaskTmp
> Task :solr:test-framework:cleanTaskTmp
> Task :solr:solrj:cleanTaskTmp
> Task :solr:solr-ref-guide:cleanTaskTmp
> Task :solr:server:cleanTaskTmp
> Task :solr:core:cleanTaskTmp
> Task :lucene:test-framework:cleanTaskTmp
> Task :lucene:suggest:cleanTaskTmp
> Task :lucene:spatial3d:cleanTaskTmp
> Task :lucene:spatial-extras:cleanTaskTmp
> Task :lucene:sandbox:cleanTaskTmp
> Task :lucene:replicator:cleanTaskTmp
> Task :lucene:queryparser:cleanTaskTmp
> Task :lucene:queries:cleanTaskTmp
> Task :lucene:monitor:cleanTaskTmp
> Task :lucene:misc:cleanTaskTmp
> Task :lucene:memory:cleanTaskTmp
> Task :lucene:luke:cleanTaskTmp
> Task :lucene:join:cleanTaskTmp
> Task :lucene:highlighter:cleanTaskTmp
> Task :lucene:grouping:cleanTaskTmp
> Task :lucene:facet:cleanTaskTmp
> Task :lucene:expressions:cleanTaskTmp
> Task :lucene:demo:cleanTaskTmp
> Task :lucene:core:cleanTaskTmp
> Task :lucene:codecs:cleanTaskTmp
> Task :lucene:classification:cleanTaskTmp
> Task :lucene:benchmark:cleanTaskTmp
> Task :cleanGradleTmp
> Task :lucene:backward-codecs:check
> Task :lucene:benchmark:check
> Task :lucene:classification:check
> Task :lucene:codecs:check
> Task :lucene:core:check
> Task :lucene:demo:check
> Task :lucene:expressions:check
> Task :lucene:facet:check
> Task :lucene:grouping:check
> Task :lucene:highlighter:check
> Task :lucene:join:check
> Task :lucene:luke:check
> Task :lucene:memory:check
> Task :lucene:misc:check
> Task :lucene:monitor:check
> Task :lucene:queries:check
> Task :lucene:queryparser:check
> Task :lucene:replicator:check
> Task :lucene:sandbox:check
> Task :lucene:spatial-extras:check
> Task :lucene:spatial3d:check
> Task :lucene:suggest:check
> Task :lucene:test-framework:check
> Task :solr:contrib:check
> Task :solr:core:check
> Task :solr:docker:check
> Task :solr:example:check
> Task :solr:packaging:check
> Task :solr:server:check

> Task :solr:solr-ref-guide:setupLazyProps
Building ref guide with:
  javadocLink -> https://docs.oracle.com/en/java/javase/11/docs/api/
  solrGuideDraftStatus -> DRAFT
  solrRootPath -> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/
  solrDocsVersion -> 9.0
  solrGuideVersionPath -> 9_0
  htmlSolrJavadocs -> https://lucene.apache.org/solr/9_0_0/
  htmlLuceneJavadocs -> https://lucene.apache.org/core/9_0_0/
  buildDate -> 2020-09-24
  buildYear -> 2020
  ivyCommonsCodec -> 1.13
  ivyDropwizardMetrics -> 4.1.5
  ivyLog4j -> 2.13.2
  ivyOpennlpTools -> 1.9.1
  ivyTika -> 1.24
  ivyZookeeper -> 3.6.2

> Task :solr:solr-ref-guide:prepareSources

> Task :solr:solr-ref-guide:buildNavDataFiles
Building up tree of all known pages
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.SecurityHelper (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-core/9.2.6.0/653601d48b38528e76840c6eac1ae46e2dae1a74/jruby-core-9.2.6.0.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jruby.util.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Looping over pages to build nav data
Creating /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content/_data/scrollnav.json
Creating /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content/_data/sidebar.json

> Task :solr:solr-ref-guide:jrubyPrepare
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/9.2.9.0/8e44ce7a1417966d89957bd766a148601e28828b/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Successfully installed ffi-1.13.1-java
Successfully installed rb-inotify-0.10.1
Successfully installed rb-fsevent-0.10.4
Successfully installed sass-listen-4.0.0
Successfully installed forwardable-extended-2.6.0
Successfully installed listen-3.2.1

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

Successfully installed sass-3.7.4
Successfully installed public_suffix-4.0.6
Successfully installed temple-0.8.2
Successfully installed asciidoctor-2.0.10
Successfully installed safe_yaml-1.0.5
Successfully installed rouge-1.11.1
Successfully installed pathutil-0.16.2
Successfully installed mercenary-0.3.6
Successfully installed liquid-4.0.3
Successfully installed kramdown-1.17.0
Successfully installed jekyll-watch-1.5.1
Successfully installed jekyll-sass-converter-1.5.2
Successfully installed colorator-1.1.0
Successfully installed addressable-2.7.0
Successfully installed concurrent-ruby-1.0.5-java
Successfully installed tilt-2.0.10
Successfully installed slim-4.0.1
Successfully installed jekyll-3.5.2
Successfully installed jekyll-asciidoc-3.0.0
25 gems installed

> Task :solr:solr-ref-guide:buildSiteJekyll
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/9.2.9.0/8e44ce7a1417966d89957bd766a148601e28828b/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Configuration file: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content/_config.yml
       Deprecation: The 'gems' configuration option has been renamed to 'plugins'. Please update your config file accordingly.
            Source: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content
       Destination: ../html-site
 Incremental build: disabled. Enable with --incremental
      Generating... 
                    done in 18.365 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

> Task :solr:solr-ref-guide:buildSite
Processed 2433 links (1750 relative) to 3530 anchors in 245 files

> Task :solr:solr-ref-guide:check
> Task :solr:solrj:check
> Task :solr:test-framework:check
> Task :solr:webapp:check
> Task :lucene:analysis:common:check
> Task :lucene:analysis:icu:check
> Task :lucene:analysis:kuromoji:check
> Task :lucene:analysis:morfologik:check
> Task :lucene:analysis:nori:check
> Task :lucene:analysis:opennlp:check
> Task :lucene:analysis:phonetic:check
> Task :lucene:analysis:smartcn:check
> Task :lucene:analysis:stempel:check
> Task :solr:contrib:analysis-extras:check
> Task :solr:contrib:analytics:check
> Task :solr:contrib:clustering:check
> Task :solr:contrib:extraction:check
> Task :solr:contrib:jaegertracer-configurator:check
> Task :solr:contrib:langid:check
> Task :solr:contrib:ltr:check
> Task :solr:contrib:prometheus-exporter:check
> Task :solr:docker:package:check
The slowest tests (exceeding 500 ms) during this run:
  253.27s TestDistributedSearch.test (:solr:core)
  192.40s TestSolrConfigHandlerCloud.test (:solr:core)
  127.24s TestDistributedStatsComponentCardinality.test (:solr:core)
  119.11s TestSolrCachePerf.testGetPutCompute (:solr:core)
  95.52s TestStressLiveNodes.testStress (:solr:core)
  81.71s DistributedFacetPivotLongTailTest.test (:solr:core)
  73.77s DistributedSpellCheckComponentTest.test (:solr:core)
  72.57s HdfsDirectoryTest.testRandomAccessWrites (:solr:core)
  67.71s PeerSyncReplicationTest.test (:solr:core)
  65.30s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection (:solr:core)

ERROR: The following test(s) have failed:
  - org.apache.solr.handler.component.CustomHighlightComponentTest.test (:solr:core)
    Test output: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.handler.component.CustomHighlightComponentTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.handler.component.CustomHighlightComponentTest" -Ptests.jvms=5 -Ptests.haltonfailure=false -Ptests.jvmargs=-XX:+UseCompressedOops -XX:+UseSerialGC -Ptests.seed=ED53A5875DD0FD37 -Ptests.badapples=false -Ptests.file.encoding=US-ASCII


Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.4.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1h 15m 56s
837 actionable tasks: 837 executed
Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2875)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2754)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2735)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3122)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to MacOSX VBOX
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at hudson.FilePath.act(FilePath.java:1070)
		at hudson.FilePath.act(FilePath.java:1059)
		at hudson.FilePath.validateAntFileMask(FilePath.java:2733)
		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
		at hudson.model.Build$BuildExecution.post2(Build.java:177)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
		at hudson.model.Run.execute(Run.java:1919)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:428)
Caused: hudson.FilePath$TunneledInterruptedException
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3124)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:375)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.validateAntFileMask(FilePath.java:2733)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
	at hudson.model.Build$BuildExecution.post2(Build.java:177)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
	at hudson.model.Run.execute(Run.java:1919)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:428)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[Java] Sleeping for 5 seconds due to JENKINS-32191...
[Java] Parsing console log (workspace: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX')
[Java] Successfully parsed console log
[Java] -> found 0 issues (skipped 0 duplicates)
[Java] Skipping post processing
[Java] No filter has been set, publishing all 0 issues
[Java] Using reference build 'Lucene-Solr-master-MacOSX #6565' to compute new, fixed, and outstanding issues
[Java] Issues delta (vs. reference build): outstanding: 0, new: 0, fixed: 0
[Java] No quality gates have been set - skipping
[Java] Health report is disabled - skipping
[Java] Created analysis result for 0 issues (found 0 new issues, fixed 0 issues)
[Java] Attaching ResultAction with ID 'java' to run 'Lucene-Solr-master-MacOSX #6566'.
[Checks API] No suitable checks publisher found.
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
[Checks API] No suitable checks publisher found.
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-12.0.2) - Build # 6565 - Still Unstable!

Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/6565/
Java: 64bit/jdk-12.0.2 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.handler.component.CustomHighlightComponentTest.test

Error Message:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:50880/solr/collection76: error processing commands

Stack Trace:
org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:50880/solr/collection76: error processing commands
	at __randomizedtesting.SeedInfo.seed([1874C501B648E637:9020FADB18B48BCF]:0)
	at org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException.create(BaseHttpSolrClient.java:67)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:664)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
	at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
	at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
	at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
	at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1170)
	at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
	at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
	at org.apache.solr.handler.component.CustomHighlightComponentTest.test(CustomHighlightComponentTest.java:152)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:567)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
	at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 8823 lines...]
  2> 241395 ERROR (qtp791209338-1042) [n:127.0.0.1:50880_solr c:collection76 s:shard1 r:core_node5 x:collection76_shard1_replica_n3 ] o.a.s.h.SolrConfigHandler ERROR:[{
  2>     "add-searchcomponent":{
  2>       "name":"customhighlight-359066317",
  2>       "class":"org.apache.solr.handler.component.CustomHighlightComponentTest$CustomHighlightComponent"},
  2>     "errorMessages":[" 'customhighlight-359066317' already exists . Do an 'update-searchcomponent' , if you want to change it "]}]
  2> 241395 INFO  (qtp791209338-1042) [n:127.0.0.1:50880_solr c:collection76 s:shard1 r:core_node5 x:collection76_shard1_replica_n3 ] o.a.s.c.S.Request [collection76_shard1_replica_n3]  webapp=/solr path=/config params={wt=javabin&version=2} status=400 QTime=16
  2> 241396 ERROR (TEST-CustomHighlightComponentTest.test-seed#[1874C501B648E637]) [     ] o.a.s.c.s.i.BaseCloudSolrClient Request to collection [collection76] failed due to (400) org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:50880/solr/collection76: error processing commands, retry=0 commError=false errorCode=400 
  2> 241397 INFO  (TEST-CustomHighlightComponentTest.test-seed#[1874C501B648E637]) [     ] o.a.s.c.s.i.BaseCloudSolrClient request was not communication error it seems
  2> 241398 INFO  (TEST-CustomHighlightComponentTest.test-seed#[1874C501B648E637]) [     ] o.a.s.SolrTestCaseJ4 ###Ending test
   >     org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException: Error from server at http://127.0.0.1:50880/solr/collection76: error processing commands
   >         at __randomizedtesting.SeedInfo.seed([1874C501B648E637:9020FADB18B48BCF]:0)
   >         at org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteExecutionException.create(BaseHttpSolrClient.java:67)
   >         at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:664)
   >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:269)
   >         at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
   >         at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:390)
   >         at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:360)
   >         at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1170)
   >         at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:931)
   >         at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:865)
   >         at org.apache.solr.handler.component.CustomHighlightComponentTest.test(CustomHighlightComponentTest.java:152)
   >         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   >         at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   >         at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   >         at java.base/java.lang.reflect.Method.invoke(Method.java:567)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
   >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   >         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   >         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   >         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   >         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   >         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
   >         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   >         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   >         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   >         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   >         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   >         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
   >         at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
   >         at java.base/java.lang.Thread.run(Thread.java:835)
  2> NOTE: reproduce with: gradlew test --tests CustomHighlightComponentTest.test -Dtests.seed=1874C501B648E637 -Dtests.slow=true -Dtests.locale=lrc -Dtests.timezone=Pacific/Bougainville -Dtests.asserts=true -Dtests.file.encoding=UTF-8
  2> 241426 INFO  (searcherExecutor-795-thread-1-processing-n:127.0.0.1:50876_solr) [n:127.0.0.1:50876_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n9]  Registered new searcher autowarm time: 0 ms
  2> 241548 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=659609075
  2> 241548 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50882_solr
  2> 241563 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 241563 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:50882_solr as DOWN
  2> 241570 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2028021496
  2> 241570 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50876_solr
  2> 241570 INFO  (zkCallback-642-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (5)
  2> 241570 INFO  (zkCallback-631-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (5)
  2> 241570 INFO  (zkCallback-645-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (5)
  2> 241570 INFO  (zkCallback-675-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (5)
  2> 241572 INFO  (zkCallback-665-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (5)
  2> 241574 INFO  (zkCallback-655-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (6) -> (5)
  2> 241574 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 241574 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:50876_solr as DOWN
  2> 241576 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2143709847
  2> 241576 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50880_solr
  2> 241578 INFO  (zkCallback-645-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
  2> 241578 INFO  (zkCallback-642-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
  2> 241592 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1927019238
  2> 241593 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50875_solr
  2> 241595 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1724411466
  2> 241595 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50877_solr
  2> 241607 INFO  (coreCloseExecutor-2121-thread-1) [n:127.0.0.1:50882_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  CLOSING SolrCore org.apache.solr.core.SolrCore@61031907
  2> 241607 INFO  (coreCloseExecutor-2121-thread-1) [n:127.0.0.1:50882_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n4 tag=SolrCore@61031907
  2> 241607 INFO  (coreCloseExecutor-2121-thread-1) [n:127.0.0.1:50882_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2100c1f1: rootName = solr_50882, domain = solr.core.collection76.shard2.replica_n4, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n4/com.codahale.metrics.MetricRegistry@299681ef
  2> 241610 INFO  (zkCallback-655-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
  2> 241610 INFO  (zkCallback-675-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (4)
  2> 241615 INFO  (zkCallback-665-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (2)
  2> 241616 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 241616 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:50880_solr as DOWN
  2> 241627 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 241627 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 241627 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:50877_solr as DOWN
  2> 241627 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:50875_solr as DOWN
  2> 241642 INFO  (zkCallback-631-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (5) -> (2)
  2> 241643 INFO  (zkCallback-642-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1)
  2> 241643 INFO  (zkCallback-675-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1)
  2> 241644 INFO  (zkCallback-655-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1)
  2> 241647 INFO  (zkCallback-645-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (1)
  2> 241661 INFO  (zkCallback-665-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
  2> 241663 INFO  (coreCloseExecutor-2123-thread-1) [n:127.0.0.1:50876_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n9]  CLOSING SolrCore org.apache.solr.core.SolrCore@fdc0a38
  2> 241672 INFO  (zkCallback-631-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
  2> 241678 INFO  (zkCallback-655-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-645-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-675-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-645-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-655-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-675-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-642-thread-3) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241680 INFO  (zkCallback-631-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241680 INFO  (zkCallback-665-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241680 INFO  (zkCallback-665-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241679 INFO  (zkCallback-642-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241699 INFO  (coreCloseExecutor-2126-thread-1) [n:127.0.0.1:50875_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n7]  CLOSING SolrCore org.apache.solr.core.SolrCore@6882a8a
  2> 241705 INFO  (zkCallback-631-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [1])
  2> 241706 INFO  (coreCloseExecutor-2126-thread-1) [n:127.0.0.1:50875_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n7 tag=SolrCore@6882a8a
  2> 241706 INFO  (coreCloseExecutor-2126-thread-1) [n:127.0.0.1:50875_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@223c2c7a: rootName = solr_50875, domain = solr.core.collection76.shard2.replica_n7, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n7/com.codahale.metrics.MetricRegistry@67d9bbeb
  2> 241729 INFO  (coreCloseExecutor-2127-thread-1) [n:127.0.0.1:50877_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@1fb82b5a
  2> 241730 INFO  (coreCloseExecutor-2127-thread-1) [n:127.0.0.1:50877_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n1 tag=SolrCore@1fb82b5a
  2> 241730 INFO  (coreCloseExecutor-2127-thread-1) [n:127.0.0.1:50877_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@9cf195a: rootName = solr_50877, domain = solr.core.collection76.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n1/com.codahale.metrics.MetricRegistry@72433efe
  2> 241843 INFO  (coreCloseExecutor-2129-thread-1) [n:127.0.0.1:50880_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@3f5ca918
  2> 241843 INFO  (coreCloseExecutor-2129-thread-1) [n:127.0.0.1:50880_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n3 tag=SolrCore@3f5ca918
  2> 241843 INFO  (coreCloseExecutor-2129-thread-1) [n:127.0.0.1:50880_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e16802e: rootName = solr_50880, domain = solr.core.collection76.shard1.replica_n3, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n3/com.codahale.metrics.MetricRegistry@2f804ed1
  2> 241888 INFO  (searcherExecutor-793-thread-1-processing-n:127.0.0.1:50877_solr) [n:127.0.0.1:50877_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n1]  Registered new searcher autowarm time: 0 ms
  2> 241892 INFO  (Thread-306) [n:127.0.0.1:50877_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n1]  CLOSING SolrCore org.apache.solr.core.SolrCore@4d244015
  2> 241895 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=536552284
  2> 241895 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:50881_solr
  2> 241901 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.c.ZkController Publish this node as DOWN...
  2> 241902 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:50881_solr as DOWN
  2> 241910 INFO  (zkCallback-645-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 241910 INFO  (zkCallback-642-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 241910 INFO  (zkCallback-665-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 241910 INFO  (zkCallback-655-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 241910 INFO  (zkCallback-675-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 241910 INFO  (zkCallback-631-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
  2> 241949 INFO  (coreCloseExecutor-2132-thread-1) [n:127.0.0.1:50881_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n11]  CLOSING SolrCore org.apache.solr.core.SolrCore@377c5afa
  2> 241949 INFO  (coreCloseExecutor-2132-thread-1) [n:127.0.0.1:50881_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n11 tag=SolrCore@377c5afa
  2> 241949 INFO  (coreCloseExecutor-2132-thread-1) [n:127.0.0.1:50881_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@618a04c4: rootName = solr_50881, domain = solr.core.collection76.shard3.replica_n11, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n11/com.codahale.metrics.MetricRegistry@56838361
  2> 242097 INFO  (zkCallback-655-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242102 INFO  (zkCallback-645-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242102 INFO  (zkCallback-631-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242105 INFO  (zkCallback-631-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242107 INFO  (zkCallback-665-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242107 INFO  (zkCallback-665-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242108 INFO  (zkCallback-675-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242109 INFO  (zkCallback-675-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242113 INFO  (zkCallback-655-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242114 INFO  (zkCallback-645-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242115 INFO  (zkCallback-642-thread-3) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242115 INFO  (zkCallback-642-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection76/state.json] for collection [collection76] has occurred - updating... (live nodes size: [0])
  2> 242807 INFO  (searcherExecutor-793-thread-1-processing-n:127.0.0.1:50877_solr) [n:127.0.0.1:50877_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n1]  Registered new searcher autowarm time: 0 ms
  2> 244255 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 244255 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 244367 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 244371 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 244371 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 244371 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678681731932094464
  2> 244378 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 244843 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 244843 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 244846 INFO  (searcherExecutor-801-thread-1-processing-n:127.0.0.1:50882_solr) [n:127.0.0.1:50882_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  Registered new searcher autowarm time: 0 ms
  2> 244853 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  CLOSING SolrCore org.apache.solr.core.SolrCore@4742bcf5
  2> 244876 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 244876 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 244886 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.u.CommitTracker Hard AutoCommit: disabled
  2> 244886 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
  2> 244924 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 244934 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 244935 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 244935 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678681732523491328
  2> 244951 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 245269 INFO  (searcherExecutor-801-thread-1-processing-n:127.0.0.1:50882_solr) [n:127.0.0.1:50882_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n4]  Registered new searcher autowarm time: 0 ms
  2> 245459 INFO  (coreCloseExecutor-2127-thread-1) [n:127.0.0.1:50877_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@1fb82b5a
  2> 245460 INFO  (Thread-306) [n:127.0.0.1:50877_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n1 tag=SolrCore@4d244015
  2> 245460 INFO  (Thread-306) [n:127.0.0.1:50877_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@39e41e34: rootName = solr_50877, domain = solr.core.collection76.shard1.replica_n1, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n1/com.codahale.metrics.MetricRegistry@72433efe
  2> 245469 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 245471 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 245474 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 245475 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678681733089722368
  2> 245494 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 245498 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf
  2> 245500 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf
  2> 245500 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000 ms
  2> 245500 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1678681733115936768
  2> 245528 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.u.DefaultSolrCoreState New IndexWriter is ready to be used.
  2> 245991 INFO  (searcherExecutor-805-thread-1-processing-n:127.0.0.1:50875_solr) [n:127.0.0.1:50875_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n7]  Registered new searcher autowarm time: 0 ms
  2> 245992 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n7]  CLOSING SolrCore org.apache.solr.core.SolrCore@1c279158
  2> 246414 INFO  (searcherExecutor-807-thread-1-processing-n:127.0.0.1:50881_solr) [n:127.0.0.1:50881_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n11]  Registered new searcher autowarm time: 0 ms
  2> 246415 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n11]  CLOSING SolrCore org.apache.solr.core.SolrCore@2ee3989d
  2> 246502 INFO  (searcherExecutor-809-thread-1-processing-n:127.0.0.1:50880_solr) [n:127.0.0.1:50880_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n3]  Registered new searcher autowarm time: 0 ms
  2> 246503 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n3]  CLOSING SolrCore org.apache.solr.core.SolrCore@29d3a96f
  2> 246783 INFO  (searcherExecutor-805-thread-1-processing-n:127.0.0.1:50875_solr) [n:127.0.0.1:50875_solr     ] o.a.s.c.SolrCore [collection76_shard2_replica_n7]  Registered new searcher autowarm time: 0 ms
  2> 247159 INFO  (searcherExecutor-807-thread-1-processing-n:127.0.0.1:50881_solr) [n:127.0.0.1:50881_solr     ] o.a.s.c.SolrCore [collection76_shard3_replica_n11]  Registered new searcher autowarm time: 0 ms
  2> 247283 INFO  (searcherExecutor-809-thread-1-processing-n:127.0.0.1:50880_solr) [n:127.0.0.1:50880_solr     ] o.a.s.c.SolrCore [collection76_shard1_replica_n3]  Registered new searcher autowarm time: 0 ms
  2> 247304 INFO  (coreCloseExecutor-2126-thread-1) [n:127.0.0.1:50875_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@6882a8a
  2> 247304 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n7 tag=SolrCore@1c279158
  2> 247305 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4c804669: rootName = solr_50875, domain = solr.core.collection76.shard2.replica_n7, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n7/com.codahale.metrics.MetricRegistry@67d9bbeb
  2> 247940 INFO  (coreCloseExecutor-2132-thread-1) [n:127.0.0.1:50881_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@377c5afa
  2> 247941 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n11 tag=SolrCore@2ee3989d
  2> 247941 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6cee969e: rootName = solr_50881, domain = solr.core.collection76.shard3.replica_n11, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n11/com.codahale.metrics.MetricRegistry@56838361
  2> 248383 INFO  (qtp97272492-1053) [n:127.0.0.1:50876_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@f3b3c1b
  2> 248396 INFO  (coreCloseExecutor-2123-thread-1) [n:127.0.0.1:50876_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard3.replica_n9 tag=SolrCore@fdc0a38
  2> 248398 INFO  (coreCloseExecutor-2123-thread-1) [n:127.0.0.1:50876_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@10878d08: rootName = solr_50876, domain = solr.core.collection76.shard3.replica_n9, service url = null, agent id = null] for registry solr.core.collection76.shard3.replica_n9/com.codahale.metrics.MetricRegistry@7b288b90
  2> 249459 INFO  (coreCloseExecutor-2121-thread-1) [n:127.0.0.1:50882_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@61031907
  2> 249462 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard2.replica_n4 tag=SolrCore@4742bcf5
  2> 249462 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@65c49d57: rootName = solr_50882, domain = solr.core.collection76.shard2.replica_n4, service url = null, agent id = null] for registry solr.core.collection76.shard2.replica_n4/com.codahale.metrics.MetricRegistry@299681ef
  2> 249998 INFO  (coreCloseExecutor-2129-thread-1) [n:127.0.0.1:50880_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@3f5ca918
  2> 249998 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection76.shard1.replica_n3 tag=SolrCore@29d3a96f
  2> 249998 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7d501034: rootName = solr_50880, domain = solr.core.collection76.shard1.replica_n3, service url = null, agent id = null] for registry solr.core.collection76.shard1.replica_n3/com.codahale.metrics.MetricRegistry@2f804ed1
  2> 255556 WARN  (jetty-closer-2108-thread-6) [     ] o.a.s.m.SolrMetricManager Could not obtain lock to modify reporters registry: solr.node
  2> 257372 WARN  (jetty-closer-2108-thread-5) [     ] o.a.s.m.SolrMetricManager Could not obtain lock to modify reporters registry: solr.node
  2> 257958 WARN  (jetty-closer-2108-thread-1) [     ] o.a.s.m.SolrMetricManager Could not obtain lock to modify reporters registry: solr.node
  2> 259522 INFO  (Thread-306) [n:127.0.0.1:50877_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@4d244015
  2> 259522 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 259522 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@261c7428: rootName = solr_50877, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@60784493
  2> 259522 INFO  (Thread-306) [n:127.0.0.1:50877_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 259543 WARN  (jetty-closer-2108-thread-2) [     ] o.a.s.m.SolrMetricManager Could not obtain lock to modify reporters registry: solr.node
  2> 259542 WARN  (Thread-306) [n:127.0.0.1:50877_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard1_replica_n1]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard1_replica_n1]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1594) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT 1c9c1509fa08ff45f8caaebafd7d90a557dd3643 [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1173) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1636) ~[main/:?]
  2> 	... 4 more
  2> 260010 WARN  (jetty-closer-2108-thread-4) [     ] o.a.s.m.SolrMetricManager Could not obtain lock to modify reporters registry: solr.node
  2> 261060 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@1c279158
  2> 261061 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 261061 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2d485aeb: rootName = solr_50875, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@60784493
  2> 261061 INFO  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 261066 WARN  (Thread-304) [n:127.0.0.1:50875_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n7]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n7]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1594) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT 1c9c1509fa08ff45f8caaebafd7d90a557dd3643 [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1173) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1636) ~[main/:?]
  2> 	... 4 more
  2> 261214 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 261214 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3ff88728: rootName = solr_50877, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@78ed6573
  2> 261283 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 261283 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@38c08fb6: rootName = solr_50881, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@60784493
  2> 261304 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard1.leader tag=SolrCore@29d3a96f
  2> 261308 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 261308 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@48881d0d: rootName = solr_50880, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@60784493
  2> 261305 INFO  (coreCloseExecutor-2123-thread-1) [n:127.0.0.1:50876_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@fdc0a38
  2> 261309 INFO  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 261309 INFO  (coreCloseExecutor-2123-thread-1) [n:127.0.0.1:50876_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 261326 WARN  (Thread-303) [n:127.0.0.1:50880_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard1_replica_n3]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard1_replica_n3]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1594) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT 1c9c1509fa08ff45f8caaebafd7d90a557dd3643 [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1173) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1636) ~[main/:?]
  2> 	... 4 more
  2> 261329 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node tag=null
  2> 261329 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@74093cd6: rootName = solr_50876, domain = solr.node, service url = null, agent id = null] for registry solr.node/com.codahale.metrics.MetricRegistry@4e056a95
  2> 261348 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard2.leader tag=SolrCore@4742bcf5
  2> 261348 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 261348 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4bc70da8: rootName = solr_50882, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@60784493
  2> 261349 INFO  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 261356 WARN  (Thread-301) [n:127.0.0.1:50882_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n4]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard2_replica_n4]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1594) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT 1c9c1509fa08ff45f8caaebafd7d90a557dd3643 [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1173) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1636) ~[main/:?]
  2> 	... 4 more
  2> 261383 INFO  (jetty-closer-2108-thread-6) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 261406 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 261406 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@48b7b7a: rootName = solr_50875, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@78ed6573
  2> 261427 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 261428 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6c5589e4: rootName = solr_50881, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@78ed6573
  2> 261428 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection76.shard3.leader tag=SolrCore@2ee3989d
  2> 261428 INFO  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.u.DirectUpdateHandler2 Committing on IndexWriter.close()  ... SKIPPED (unnecessary).
  2> 261431 WARN  (Thread-302) [n:127.0.0.1:50881_solr     ] o.a.s.c.ZkController listener throws error
  2>           => org.apache.solr.common.SolrException: Unable to reload core [collection76_shard3_replica_n11]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675)
  2> org.apache.solr.common.SolrException: Unable to reload core [collection76_shard3_replica_n11]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1675) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1594) ~[main/:?]
  2> 	at org.apache.solr.core.SolrCore.lambda$getConfListener$20(SolrCore.java:3110) ~[main/:?]
  2> 	at org.apache.solr.cloud.ZkController.lambda$fireEventListeners$14(ZkController.java:2500) [main/:9.0.0-SNAPSHOT 1c9c1509fa08ff45f8caaebafd7d90a557dd3643 [snapshot build, details omitted]]
  2> 	at java.lang.Thread.run(Thread.java:835) [?:?]
  2> Caused by: java.lang.IllegalStateException: This CoreContainer has been closed
  2> 	at org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:1173) ~[main/:?]
  2> 	at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:1636) ~[main/:?]
  2> 	... 4 more
  2> 261454 INFO  (jetty-closer-2108-thread-5) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 261466 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 261467 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@62f8047d: rootName = solr_50880, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@78ed6573
  2> 261515 INFO  (jetty-closer-2108-thread-6) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@1ea851f9{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:۰}
  2> 261517 INFO  (jetty-closer-2108-thread-6) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@57cbd17d{/solr,null,UNAVAILABLE}
  2> 261518 INFO  (jetty-closer-2108-thread-6) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 261555 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 261555 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b31ccc1: rootName = solr_50882, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@78ed6573
  2> 261593 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 261632 INFO  (closeThreadPool-2152-thread-2) [     ] o.a.s.c.Overseer Overseer (id=72073072877371403-127.0.0.1:50881_solr-n_0000000000) closing
  2> 261637 INFO  (OverseerStateUpdate-72073072877371403-127.0.0.1:50881_solr-n_0000000000) [n:127.0.0.1:50881_solr     ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:50881_solr
  2> 261656 INFO  (closeThreadPool-2152-thread-1) [     ] o.a.s.c.Overseer Overseer (id=72073072877371403-127.0.0.1:50881_solr-n_0000000000) closing
  2> 261686 INFO  (jetty-closer-2108-thread-5) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@4ac414c4{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:۰}
  2> 261692 INFO  (jetty-closer-2108-thread-5) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@47fb91ef{/solr,null,UNAVAILABLE}
  2> 261698 INFO  (jetty-closer-2108-thread-5) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 261801 INFO  (jetty-closer-2108-thread-1) [     ] o.a.s.c.Overseer Overseer (id=72073072877371403-127.0.0.1:50881_solr-n_0000000000) closing
  2> 261818 INFO  (jetty-closer-2108-thread-1) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@c99a0f4{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:۰}
  2> 261829 INFO  (jetty-closer-2108-thread-1) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@659ea4ff{/solr,null,UNAVAILABLE}
  2> 261834 INFO  (jetty-closer-2108-thread-1) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 261860 INFO  (jetty-closer-2108-thread-4) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 261884 INFO  (jetty-closer-2108-thread-2) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 261887 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm tag=null
  2> 261888 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6c73397f: rootName = solr_50876, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm/com.codahale.metrics.MetricRegistry@60784493
  2> 261901 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty tag=null
  2> 261902 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2c33368d: rootName = solr_50876, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty/com.codahale.metrics.MetricRegistry@78ed6573
  2> 261904 INFO  (jetty-closer-2108-thread-3) [     ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster tag=null
  2> 261977 WARN  (jetty-launcher-604-thread-1-SendThread(127.0.0.1:50867)) [     ] o.a.z.ClientCnxn An exception was thrown while closing send thread for session 0x1000e13f2ba000e.
  2>           => EndOfStreamException: Unable to read additional data from server sessionid 0x1000e13f2ba000e, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
  2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x1000e13f2ba000e, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275) [zookeeper-3.6.2.jar:3.6.2]
  2> 262045 WARN  (jetty-launcher-604-thread-6-SendThread(127.0.0.1:50867)) [     ] o.a.z.ClientCnxn An exception was thrown while closing send thread for session 0x1000e13f2ba000a.
  2>           => EndOfStreamException: Unable to read additional data from server sessionid 0x1000e13f2ba000a, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77)
  2> org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x1000e13f2ba000a, likely server has closed socket
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) ~[zookeeper-3.6.2.jar:3.6.2]
  2> 	at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1275) [zookeeper-3.6.2.jar:3.6.2]
  2> 262059 INFO  (jetty-closer-2108-thread-4) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@21ef1eef{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:۰}
  2> 262063 INFO  (jetty-closer-2108-thread-4) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@5cd8d6a4{/solr,null,UNAVAILABLE}
  2> 262067 INFO  (jetty-closer-2108-thread-4) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 262177 INFO  (jetty-closer-2108-thread-2) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@1416f315{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:۰}
  2> 262185 INFO  (jetty-closer-2108-thread-2) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@a963f7b{/solr,null,UNAVAILABLE}
  2> 262191 INFO  (jetty-closer-2108-thread-2) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 262248 INFO  (jetty-closer-2108-thread-3) [     ] o.e.j.s.AbstractConnector Stopped ServerConnector@b0bbc65{HTTP/1.1, (http/1.1, h2c)}{127.0.0.1:۰}
  2> 262255 INFO  (jetty-closer-2108-thread-3) [     ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7d6453b2{/solr,null,UNAVAILABLE}
  2> 262256 INFO  (jetty-closer-2108-thread-3) [     ] o.e.j.s.session node0 Stopped scavenging
  2> 262262 INFO  (SUITE-CustomHighlightComponentTest-seed#[1874C501B648E637]-worker) [     ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
  2> 262493 WARN  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Watch limit violations: 
  2> Maximum concurrent create/delete watches above limit:
  2> 
  2> 	12	/solr/configs/conf
  2> 	7	/solr/aliases.json
  2> 	7	/solr/clusterprops.json
  2> 	6	/solr/packages.json
  2> 	6	/solr/security.json
  2> 	5	/solr/collections/collection76/terms/shard1
  2> 	5	/solr/collections/collection76/terms/shard2
  2> 	5	/solr/collections/collection76/terms/shard3
  2> 
  2> Maximum concurrent data watches above limit:
  2> 
  2> 	66	/solr/collections/collection76/state.json
  2> 
  2> Maximum concurrent children watches above limit:
  2> 
  2> 	54	/solr/live_nodes
  2> 	34	/solr/overseer/queue
  2> 	14	/solr/collections
  2> 	3	/solr/overseer/collection-queue-work
  2> 
  2> 262564 INFO  (SUITE-CustomHighlightComponentTest-seed#[1874C501B648E637]-worker) [     ] o.a.s.c.ZkTestServer waitForServerDown: 127.0.0.1:50867
  2> 262564 INFO  (SUITE-CustomHighlightComponentTest-seed#[1874C501B648E637]-worker) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:50867
  2> 262565 INFO  (SUITE-CustomHighlightComponentTest-seed#[1874C501B648E637]-worker) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 50867
  2> NOTE: leaving temporary files on disk at: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/tmp/tests-tmp/solr.handler.component.CustomHighlightComponentTest_1874C501B648E637-001
  2> Sep 24, 2020 2:47:51 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
  2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
  2> NOTE: test params are: codec=Asserting(Lucene87), sim=Asserting(RandomSimilarity(queryNorm=true): {}), locale=lrc, timezone=Pacific/Bougainville
  2> NOTE: Mac OS X 10.14.6 x86_64/AdoptOpenJDK 12.0.2 (64-bit)/cpus=6,threads=1,free=100032904,total=341639168
  2> NOTE: All tests run in this JVM: [V2StandaloneTest, XsltUpdateRequestHandlerTest, TestHdfsBackupRestoreCore, CoreAdminRequestStatusTest, ShowFileRequestHandlerTest, DaemonStreamApiTest, ZookeeperReadAPITest, IndexSizeEstimatorTest, TestCollectionAPIs, TestPivotHelperCode, SpellCheckComponentTest, QueryElevationComponentTest, CustomHighlightComponentTest]

WARNING: Test org.apache.solr.TestDistributedSearch wrote 27,022,340 bytes of output.
WARNING: Test org.apache.solr.schema.DocValuesTest wrote 12,185,244 bytes of output.
WARNING: Test org.apache.solr.TestRandomFaceting wrote 11,349,543 bytes of output.
WARNING: Test org.apache.solr.search.facet.TestCloudJSONFacetSKG wrote 35,965,582 bytes of output.
WARNING: Test org.apache.solr.search.facet.TestJsonFacets wrote 28,449,939 bytes of output.
WARNING: Test org.apache.solr.schema.TestCloudSchemaless wrote 24,091,615 bytes of output.
:solr:core:test (FAILURE): 4580 test(s), 1 failure(s), 205 skipped

4580 tests completed, 1 failed, 205 skipped
There were failing tests. See the results at: file:///Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/test-results/test/

> Task :solr:server:test NO-SOURCE
> Task :solr:solr-ref-guide:test NO-SOURCE

> Task :solr:solrj:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:solrj:test (SUCCESS): 1665 test(s), 24 skipped

> Task :solr:test-framework:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:test-framework:test (SUCCESS): 9 test(s)

> Task :solr:webapp:test NO-SOURCE

> Task :lucene:analysis:common:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:common:test (SUCCESS): 1639 test(s), 9 skipped

> Task :lucene:analysis:icu:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:icu:test (SUCCESS): 149 test(s), 11 skipped

> Task :lucene:analysis:kuromoji:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:kuromoji:test (SUCCESS): 159 test(s), 7 skipped

> Task :lucene:analysis:morfologik:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:morfologik:test (SUCCESS): 18 test(s)

> Task :lucene:analysis:nori:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:nori:test (SUCCESS): 69 test(s), 4 skipped

> Task :lucene:analysis:opennlp:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:opennlp:test (SUCCESS): 26 test(s)

> Task :lucene:analysis:phonetic:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:phonetic:test (SUCCESS): 41 test(s)

> Task :lucene:analysis:smartcn:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:smartcn:test (SUCCESS): 21 test(s)

> Task :lucene:analysis:stempel:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:lucene:analysis:stempel:test (SUCCESS): 15 test(s)

> Task :solr:contrib:analysis-extras:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:analysis-extras:test (SUCCESS): 30 test(s)

> Task :solr:contrib:analytics:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:analytics:test (SUCCESS): 472 test(s)

> Task :solr:contrib:clustering:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:clustering:test (SUCCESS): 30 test(s)

> Task :solr:contrib:extraction:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:extraction:test (SUCCESS): 20 test(s)

> Task :solr:contrib:jaegertracer-configurator:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:jaegertracer-configurator:test (SUCCESS): 3 test(s)

> Task :solr:contrib:langid:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:langid:test (SUCCESS): 43 test(s)

> Task :solr:contrib:ltr:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:ltr:test (SUCCESS): 127 test(s), 1 skipped

> Task :solr:contrib:prometheus-exporter:test
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
:solr:contrib:prometheus-exporter:test (SUCCESS): 17 test(s), 5 skipped

> Task :solr:contrib:prometheus-exporter:cleanTaskTmp
> Task :solr:contrib:ltr:cleanTaskTmp
> Task :solr:contrib:langid:cleanTaskTmp
> Task :solr:contrib:jaegertracer-configurator:cleanTaskTmp
> Task :solr:contrib:extraction:cleanTaskTmp
> Task :solr:contrib:clustering:cleanTaskTmp
> Task :solr:contrib:analytics:cleanTaskTmp
> Task :solr:contrib:analysis-extras:cleanTaskTmp
> Task :lucene:analysis:stempel:cleanTaskTmp
> Task :lucene:analysis:smartcn:cleanTaskTmp
> Task :lucene:analysis:phonetic:cleanTaskTmp
> Task :lucene:analysis:opennlp:cleanTaskTmp
> Task :lucene:analysis:nori:cleanTaskTmp
> Task :lucene:analysis:morfologik:cleanTaskTmp
> Task :lucene:analysis:kuromoji:cleanTaskTmp
> Task :lucene:analysis:icu:cleanTaskTmp
> Task :lucene:analysis:common:cleanTaskTmp
> Task :solr:webapp:cleanTaskTmp
> Task :solr:test-framework:cleanTaskTmp
> Task :solr:solrj:cleanTaskTmp
> Task :solr:solr-ref-guide:cleanTaskTmp
> Task :solr:server:cleanTaskTmp
> Task :solr:core:cleanTaskTmp
> Task :lucene:test-framework:cleanTaskTmp
> Task :lucene:suggest:cleanTaskTmp
> Task :lucene:spatial3d:cleanTaskTmp
> Task :lucene:spatial-extras:cleanTaskTmp
> Task :lucene:sandbox:cleanTaskTmp
> Task :lucene:replicator:cleanTaskTmp
> Task :lucene:queryparser:cleanTaskTmp
> Task :lucene:queries:cleanTaskTmp
> Task :lucene:monitor:cleanTaskTmp
> Task :lucene:misc:cleanTaskTmp
> Task :lucene:memory:cleanTaskTmp
> Task :lucene:luke:cleanTaskTmp
> Task :lucene:join:cleanTaskTmp
> Task :lucene:highlighter:cleanTaskTmp
> Task :lucene:grouping:cleanTaskTmp
> Task :lucene:facet:cleanTaskTmp
> Task :lucene:expressions:cleanTaskTmp
> Task :lucene:demo:cleanTaskTmp
> Task :lucene:core:cleanTaskTmp
> Task :lucene:codecs:cleanTaskTmp
> Task :lucene:classification:cleanTaskTmp
> Task :lucene:benchmark:cleanTaskTmp
> Task :cleanGradleTmp
> Task :lucene:backward-codecs:check
> Task :lucene:benchmark:check
> Task :lucene:classification:check
> Task :lucene:codecs:check
> Task :lucene:core:check
> Task :lucene:demo:check
> Task :lucene:expressions:check
> Task :lucene:facet:check
> Task :lucene:grouping:check
> Task :lucene:highlighter:check
> Task :lucene:join:check
> Task :lucene:luke:check
> Task :lucene:memory:check
> Task :lucene:misc:check
> Task :lucene:monitor:check
> Task :lucene:queries:check
> Task :lucene:queryparser:check
> Task :lucene:replicator:check
> Task :lucene:sandbox:check
> Task :lucene:spatial-extras:check
> Task :lucene:spatial3d:check
> Task :lucene:suggest:check
> Task :lucene:test-framework:check
> Task :solr:contrib:check
> Task :solr:core:check
> Task :solr:docker:check
> Task :solr:example:check
> Task :solr:packaging:check
> Task :solr:server:check

> Task :solr:solr-ref-guide:setupLazyProps
Building ref guide with:
  javadocLink -> https://docs.oracle.com/en/java/javase/11/docs/api/
  solrGuideDraftStatus -> DRAFT
  solrRootPath -> /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/
  solrDocsVersion -> 9.0
  solrGuideVersionPath -> 9_0
  htmlSolrJavadocs -> https://lucene.apache.org/solr/9_0_0/
  htmlLuceneJavadocs -> https://lucene.apache.org/core/9_0_0/
  buildDate -> 2020-09-24
  buildYear -> 2020
  ivyCommonsCodec -> 1.13
  ivyDropwizardMetrics -> 4.1.5
  ivyLog4j -> 2.13.2
  ivyOpennlpTools -> 1.9.1
  ivyTika -> 1.24
  ivyZookeeper -> 3.6.2

> Task :solr:solr-ref-guide:prepareSources

> Task :solr:solr-ref-guide:buildNavDataFiles
Building up tree of all known pages
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.util.SecurityHelper (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-core/9.2.6.0/653601d48b38528e76840c6eac1ae46e2dae1a74/jruby-core-9.2.6.0.jar) to field java.lang.reflect.Field.modifiers
WARNING: Please consider reporting this to the maintainers of org.jruby.util.SecurityHelper
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Looping over pages to build nav data
Creating /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content/_data/scrollnav.json
Creating /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content/_data/sidebar.json

> Task :solr:solr-ref-guide:jrubyPrepare
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/9.2.9.0/8e44ce7a1417966d89957bd766a148601e28828b/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Successfully installed ffi-1.13.1-java
Successfully installed rb-inotify-0.10.1
Successfully installed rb-fsevent-0.10.4
Successfully installed sass-listen-4.0.0
Successfully installed forwardable-extended-2.6.0
Successfully installed listen-3.2.1

Ruby Sass has reached end-of-life and should no longer be used.

* If you use Sass as a command-line tool, we recommend using Dart Sass, the new
  primary implementation: https://sass-lang.com/install

* If you use Sass as a plug-in for a Ruby web framework, we recommend using the
  sassc gem: https://github.com/sass/sassc-ruby#readme

* For more details, please refer to the Sass blog:
  https://sass-lang.com/blog/posts/7828841

Successfully installed sass-3.7.4
Successfully installed public_suffix-4.0.6
Successfully installed temple-0.8.2
Successfully installed asciidoctor-2.0.10
Successfully installed safe_yaml-1.0.5
Successfully installed rouge-1.11.1
Successfully installed pathutil-0.16.2
Successfully installed mercenary-0.3.6
Successfully installed liquid-4.0.3
Successfully installed kramdown-1.17.0
Successfully installed jekyll-watch-1.5.1
Successfully installed jekyll-sass-converter-1.5.2
Successfully installed colorator-1.1.0
Successfully installed addressable-2.7.0
Successfully installed concurrent-ruby-1.0.5-java
Successfully installed tilt-2.0.10
Successfully installed slim-4.0.1
Successfully installed jekyll-3.5.2
Successfully installed jekyll-asciidoc-3.0.0
25 gems installed

> Task :solr:solr-ref-guide:buildSiteJekyll
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/Users/jenkins/.gradle/caches/modules-2/files-2.1/org.jruby/jruby-complete/9.2.9.0/8e44ce7a1417966d89957bd766a148601e28828b/jruby-complete-9.2.9.0.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
Configuration file: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content/_config.yml
       Deprecation: The 'gems' configuration option has been renamed to 'plugins'. Please update your config file accordingly.
            Source: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/solr-ref-guide/build/content
       Destination: ../html-site
 Incremental build: disabled. Enable with --incremental
      Generating... 
                    done in 18.708 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

> Task :solr:solr-ref-guide:buildSite
Processed 2433 links (1750 relative) to 3530 anchors in 245 files

> Task :solr:solr-ref-guide:check
> Task :solr:solrj:check
> Task :solr:test-framework:check
> Task :solr:webapp:check
> Task :lucene:analysis:common:check
> Task :lucene:analysis:icu:check
> Task :lucene:analysis:kuromoji:check
> Task :lucene:analysis:morfologik:check
> Task :lucene:analysis:nori:check
> Task :lucene:analysis:opennlp:check
> Task :lucene:analysis:phonetic:check
> Task :lucene:analysis:smartcn:check
> Task :lucene:analysis:stempel:check
> Task :solr:contrib:analysis-extras:check
> Task :solr:contrib:analytics:check
> Task :solr:contrib:clustering:check
> Task :solr:contrib:extraction:check
> Task :solr:contrib:jaegertracer-configurator:check
> Task :solr:contrib:langid:check
> Task :solr:contrib:ltr:check
> Task :solr:contrib:prometheus-exporter:check
> Task :solr:docker:package:check
The slowest tests (exceeding 500 ms) during this run:
  244.11s TestDistributedSearch.test (:solr:core)
  205.42s TestSolrConfigHandlerCloud.test (:solr:core)
  145.47s TestSolrCachePerf.testGetPutCompute (:solr:core)
  117.04s TestDistributedStatsComponentCardinality.test (:solr:core)
  84.90s DistributedFacetPivotLongTailTest.test (:solr:core)
  80.25s TestStressLiveNodes.testStress (:solr:core)
  80.04s TestInPlaceUpdatesDistrib.test (:solr:core)
  76.84s DistributedSpellCheckComponentTest.test (:solr:core)
  71.56s LeaderElectionIntegrationTest.testSimpleSliceLeaderElection (:solr:core)
  68.66s CheckHdfsIndexTest.testDeletedDocs (:solr:core)

ERROR: The following test(s) have failed:
  - org.apache.solr.handler.component.CustomHighlightComponentTest.test (:solr:core)
    Test output: /Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/build/test-results/test/outputs/OUTPUT-org.apache.solr.handler.component.CustomHighlightComponentTest.txt
    Reproduce with: gradlew :solr:core:test --tests "org.apache.solr.handler.component.CustomHighlightComponentTest" -Ptests.jvms=5 -Ptests.haltonfailure=false -Ptests.jvmargs=-XX:-UseCompressedOops -XX:+UseConcMarkSweepGC -Ptests.seed=1874C501B648E637 -Ptests.badapples=false -Ptests.file.encoding=UTF-8


Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.4.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 1h 21m 22s
837 actionable tasks: 837 executed
Build step 'Invoke Gradle script' changed build result to SUCCESS
Archiving artifacts
java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2875)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2754)
	at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2735)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3122)
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to MacOSX VBOX
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1800)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at hudson.FilePath.act(FilePath.java:1070)
		at hudson.FilePath.act(FilePath.java:1059)
		at hudson.FilePath.validateAntFileMask(FilePath.java:2733)
		at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
		at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
		at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
		at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
		at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
		at hudson.model.Build$BuildExecution.post2(Build.java:177)
		at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
		at hudson.model.Run.execute(Run.java:1919)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:97)
		at hudson.model.Executor.run(Executor.java:428)
Caused: hudson.FilePath$TunneledInterruptedException
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3124)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:375)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:73)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1059)
	at hudson.FilePath.validateAntFileMask(FilePath.java:2733)
	at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:270)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:80)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:804)
	at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:753)
	at hudson.model.Build$BuildExecution.post2(Build.java:177)
	at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:698)
	at hudson.model.Run.execute(Run.java:1919)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:97)
	at hudson.model.Executor.run(Executor.java:428)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[Java] Sleeping for 5 seconds due to JENKINS-32191...
[Java] Parsing console log (workspace: '/Users/jenkins/workspace/Lucene-Solr-master-MacOSX')
[Java] Successfully parsed console log
[Java] -> found 0 issues (skipped 0 duplicates)
[Java] Skipping post processing
[Java] No filter has been set, publishing all 0 issues
[Java] Using reference build 'Lucene-Solr-master-MacOSX #6564' to compute new, fixed, and outstanding issues
[Java] Issues delta (vs. reference build): outstanding: 0, new: 0, fixed: 0
[Java] No quality gates have been set - skipping
[Java] Health report is disabled - skipping
[Java] Created analysis result for 0 issues (found 0 new issues, fixed 0 issues)
[Java] Attaching ResultAction with ID 'java' to run 'Lucene-Solr-master-MacOSX #6565'.
[Checks API] No suitable checks publisher found.
Recording test results
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
[Checks API] No suitable checks publisher found.
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)