You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/12/09 15:14:26 UTC
[JENKINS] Lucene-Solr-http2-Linux (64bit/jdk-10.0.1) - Build # 54 -
Still unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/54/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseG1GC
2 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test: 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test] at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method) at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test:
1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([CCBEAFBF9E267216]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test] at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method) at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292) at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
at __randomizedtesting.SeedInfo.seed([CCBEAFBF9E267216]:0)
Build Log:
[...truncated 16228 lines...]
[junit4] Suite: org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-solrj/test/J2/temp/solr.client.solrj.embedded.SolrExampleXMLHttp2Test_CCBEAFBF9E267216-001/init-core-data-001
[junit4] 1> 15:06:27.326 [SUITE-SolrExampleXMLHttp2Test-seed#[CCBEAFBF9E267216]-worker] ERROR org.apache.solr.util.StartupLoggingUtils - Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 1> {"responseHeader":{
[junit4] 1> "status":0,
[junit4] 1> "QTime":22}}
[junit4] 1> 15:06:38.558 [qtp1635395382-255] ERROR org.apache.solr.handler.RequestHandlerBase - org.apache.solr.common.SolrException: stats local param of facet.pivotmay not include tags separated by a comma - please use a common tag on all stats.field params you wish to compute under this pivot
[junit4] 1> at org.apache.solr.handler.component.PivotFacetProcessor.getTaggedStatsFields(PivotFacetProcessor.java:256)
[junit4] 1> at org.apache.solr.handler.component.PivotFacetProcessor.process(PivotFacetProcessor.java:122)
[junit4] 1> at org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:279)
[junit4] 1> at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:298)
[junit4] 1> at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
[junit4] 1> at org.apache.solr.core.SolrCore.execute(SolrCore.java:2549)
[junit4] 1> at org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:710)
[junit4] 1> at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:515)
[junit4] 1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:396)
[junit4] 1> at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:342)
[junit4] 1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
[junit4] 1> at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:164)
[junit4] 1> at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
[junit4] 1> at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
[junit4] 1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
[junit4] 1> at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
[junit4] 1> at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
[junit4] 1> at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
[junit4] 1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
[junit4] 1> at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
[junit4] 1> at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
[junit4] 1> at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
[junit4] 1> at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
[junit4] 1> at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
[junit4] 1> at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:753)
[junit4] 1> at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
[junit4] 1> at org.eclipse.jetty.server.Server.handle(Server.java:502)
[junit4] 1> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
[junit4] 1> at org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:305)
[junit4] 1> at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
[junit4] 1> at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
[junit4] 1> at java.base/java.lang.Thread.run(Thread.java:844)
[junit4] 1>
[junit4] 2> Dec 09, 2018 3:06:39 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
[junit4] 2> Dec 09, 2018 3:06:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test:
[junit4] 2> 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
[junit4] 2> at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] 2> at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] 2> at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
[junit4] 2> at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
[junit4] 2> Dec 09, 2018 3:06:42 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> INFO: Starting to interrupt leaked threads:
[junit4] 2> 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
[junit4] 2> Dec 09, 2018 3:06:45 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
[junit4] 2> SEVERE: There are still zombie threads that couldn't be terminated:
[junit4] 2> 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
[junit4] 2> at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] 2> at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] 2> at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
[junit4] 2> at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
[junit4] 2> at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
[junit4] 2> at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=743, maxMBSortInHeap=7.5102267511068765, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@e136bc), locale=ne, timezone=America/Argentina/Buenos_Aires
[junit4] 2> NOTE: Linux 4.15.0-39-generic amd64/Oracle Corporation 10.0.1 (64-bit)/cpus=8,threads=2,free=361811328,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [Http2SolrClientTest, FacetFieldTest, SolrExampleXMLHttp2Test]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=SolrExampleXMLHttp2Test -Dtests.seed=CCBEAFBF9E267216 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ne -Dtests.timezone=America/Argentina/Buenos_Aires -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J2 | SolrExampleXMLHttp2Test (suite) <<<
[junit4] > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test:
[junit4] > 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
[junit4] > at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] > at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
[junit4] > at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
[junit4] > at __randomizedtesting.SeedInfo.seed([CCBEAFBF9E267216]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4] > 1) Thread[id=682, name=h2sc-qtp-[java.base/java.lang.Thread.getStackTrace(Thread.java:1654), org.apache.solr.client.solrj.impl.Http2SolrClient.createHttpClient(Http2SolrClient.java:173), org.apache.solr.client.solrj.impl.Http2SolrClient.<init>(Http2SolrClient.java:138), org.apache.solr.client.solrj.impl.Http2SolrClient$Builder.build(Http2SolrClient.java:807), org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test.createNewSolrClient(SolrExampleXMLHttp2Test.java:37), org.apache.solr.SolrJettyTestBase.getSolrClient(SolrJettyTestBase.java:138), org.apache.solr.client.solrj.SolrExampleTests.emptyCollection(SolrExampleTests.java:102), jdk.internal.reflect.GeneratedMethodAccessor9.invoke(Unknown Source), java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43), java.base/java.lang.reflect.Method.invoke(Method.java:564), com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742), com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:969), com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817), com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468), com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944), com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830), com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880), com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57), org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53), org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47), org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64), org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54), com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36), com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368), java.base/java.lang.Thread.run(Thread.java:844)]-682, state=TIMED_WAITING, group=TGRP-SolrExampleXMLHttp2Test]
[junit4] > at java.base@10.0.1/jdk.internal.misc.Unsafe.park(Native Method)
[junit4] > at java.base@10.0.1/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
[junit4] > at java.base@10.0.1/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2205)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
[junit4] > at app//org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
[junit4] > at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
[junit4] > at java.base@10.0.1/java.lang.Thread.run(Thread.java:844)
[junit4] > at __randomizedtesting.SeedInfo.seed([CCBEAFBF9E267216]:0)
[junit4] Completed [19/184 (1!)] on J2 in 17.89s, 38 tests, 2 errors <<< FAILURES!
[...truncated 41891 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/54/consoleText
[repro] Revision: cc55082f217f958bd1f2d04c68e81ac21d8e938d
[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC"
[repro] Repro line: ant test -Dtestcase=SolrExampleXMLHttp2Test -Dtests.seed=CCBEAFBF9E267216 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ne -Dtests.timezone=America/Argentina/Buenos_Aires -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[repro] ant clean
[...truncated 6 lines...]
[repro] Test suites by module:
[repro] solr/solrj
[repro] SolrExampleXMLHttp2Test
[repro] ant compile-test
[...truncated 2716 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.SolrExampleXMLHttp2Test" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseG1GC" -Dtests.seed=CCBEAFBF9E267216 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ne -Dtests.timezone=America/Argentina/Buenos_Aires -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[...truncated 260 lines...]
[repro] Failures:
[repro] 0/5 failed: org.apache.solr.client.solrj.embedded.SolrExampleXMLHttp2Test
[repro] Exiting with code 0
[...truncated 40 lines...]
[JENKINS] Lucene-Solr-http2-Linux (64bit/jdk-11) - Build # 56 -
Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/56/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseParallelGC
19 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
Error Message:
74 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest: 1) Thread[id=1071, name=qtp154888852-1071, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 2) Thread[id=1001, name=qtp1645118601-1001, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 3) Thread[id=1279, name=updateExecutor-188-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 4) Thread[id=1253, name=Scheduler-2089236759, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 5) Thread[id=977, name=qtp2079431130-977, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 6) Thread[id=1037, name=qtp1242769164-1037, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 7) Thread[id=1213, name=coreContainerWorkExecutor-213-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 8) Thread[id=1255, name=Scheduler-1208377519, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 9) Thread[id=923, name=qtp66645842-923, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 10) Thread[id=1002, name=qtp1645118601-1002-acceptor-0@cbb0765-ServerConnector@4695d09b{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:41353}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 11) Thread[id=1229, name=SolrRrdBackendFactory-242-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 12) Thread[id=1275, name=updateExecutor-231-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 13) Thread[id=924, name=qtp66645842-924, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 14) Thread[id=1000, name=qtp1645118601-1000, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 15) Thread[id=928, name=qtp66645842-928, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 16) Thread[id=1045, name=qtp1242769164-1045-acceptor-0@74e7ce15-ServerConnector@7b69117d{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:44499}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 17) Thread[id=972, name=qtp2079431130-972, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 18) Thread[id=999, name=qtp1645118601-999, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 19) Thread[id=998, name=qtp1645118601-998, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 20) Thread[id=1077, name=qtp154888852-1077, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 21) Thread[id=995, name=coreContainerWorkExecutor-202-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 22) Thread[id=1234, name=coreContainerWorkExecutor-218-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 23) Thread[id=1232, name=coreContainerWorkExecutor-227-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 24) Thread[id=974, name=qtp2079431130-974-acceptor-0@14fbca69-ServerConnector@5fde07f3{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:43939}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 25) Thread[id=1075, name=qtp154888852-1075-acceptor-0@5d6ed29a-ServerConnector@54088673{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:43629}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 26) Thread[id=1006, name=Scheduler-916823175, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 27) Thread[id=1043, name=qtp1242769164-1043, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 28) Thread[id=931, name=Scheduler-1974536469, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 29) Thread[id=980, name=Scheduler-1939903640, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 30) Thread[id=1090, name=coreContainerWorkExecutor-207-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 31) Thread[id=1072, name=qtp154888852-1072, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 32) Thread[id=1256, name=Scheduler-986485729, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 33) Thread[id=993, name=MetricsHistoryHandler-210-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 34) Thread[id=1257, name=Scheduler-187575607, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 35) Thread[id=1005, name=qtp1645118601-1005, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 36) Thread[id=926, name=qtp66645842-926, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 37) Thread[id=1073, name=qtp154888852-1073, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 38) Thread[id=1079, name=qtp154888852-1079, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 39) Thread[id=994, name=Thread-532, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/java.lang.Object.wait(Native Method) at java.base@11/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 40) Thread[id=1233, name=Thread-688, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/java.lang.Object.wait(Native Method) at java.base@11/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 41) Thread[id=973, name=qtp2079431130-973, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 42) Thread[id=929, name=qtp66645842-929, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 43) Thread[id=1003, name=qtp1645118601-1003, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 44) Thread[id=1039, name=qtp1242769164-1039, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 45) Thread[id=1004, name=qtp1645118601-1004, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 46) Thread[id=1041, name=qtp1242769164-1041, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 47) Thread[id=927, name=qtp66645842-927-acceptor-0@58f24658-ServerConnector@3b602680{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:43429}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533) at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285) at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385) at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 48) Thread[id=1086, name=Scheduler-934184420, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 49) Thread[id=992, name=SolrRrdBackendFactory-209-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 50) Thread[id=1047, name=qtp1242769164-1047, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 51) Thread[id=1051, name=qtp1242769164-1051, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 52) Thread[id=1231, name=Thread-687, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/java.lang.Object.wait(Native Method) at java.base@11/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 53) Thread[id=930, name=qtp66645842-930, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 54) Thread[id=1225, name=SolrRrdBackendFactory-240-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 55) Thread[id=1179, name=SolrRrdBackendFactory-235-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 56) Thread[id=1053, name=Scheduler-1069203947, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 57) Thread[id=975, name=qtp2079431130-975, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 58) Thread[id=1074, name=qtp154888852-1074, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 59) Thread[id=1089, name=Thread-598, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/java.lang.Object.wait(Native Method) at java.base@11/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 60) Thread[id=1228, name=MetricsHistoryHandler-241-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 61) Thread[id=925, name=qtp66645842-925, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 62) Thread[id=1278, name=updateExecutor-188-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 63) Thread[id=1049, name=qtp1242769164-1049, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 64) Thread[id=1230, name=MetricsHistoryHandler-243-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 65) Thread[id=1254, name=Scheduler-1120265577, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 66) Thread[id=1081, name=qtp154888852-1081, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 67) Thread[id=1068, name=SolrRrdBackendFactory-220-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 68) Thread[id=976, name=qtp2079431130-976, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720) at java.base@11/java.lang.Thread.run(Thread.java:834) 69) Thread[id=1069, name=MetricsHistoryHandler-221-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 70) Thread[id=1276, name=recoveryExecutor-240-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462) at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361) at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 71) Thread[id=971, name=qtp2079431130-971, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834) 72) Thread[id=1181, name=MetricsHistoryHandler-236-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182) at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 73) Thread[id=1208, name=Thread-686, state=WAITING, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/java.lang.Object.wait(Native Method) at java.base@11/java.lang.Object.wait(Object.java:328) at app//org.apache.solr.core.CloserThread.run(CoreContainer.java:1902) 74) Thread[id=970, name=qtp2079431130-970, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest] at java.base@11/sun.nio.ch.EPoll.wait(Native Method) at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120) at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124) at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423) at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source) at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765) at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 74 threads leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
1) Thread[id=1071, name=qtp154888852-1071, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
2) Thread[id=1001, name=qtp1645118601-1001, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
3) Thread[id=1279, name=updateExecutor-188-thread-2, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
4) Thread[id=1253, name=Scheduler-2089236759, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=977, name=qtp2079431130-977, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at app//org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
at java.base@11/java.lang.Thread.run(Thread.java:834)
6) Thread[id=1037, name=qtp1242769164-1037, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
7) Thread[id=1213, name=coreContainerWorkExecutor-213-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:462)
at java.base@11/java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:361)
at java.base@11/java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:937)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1053)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
8) Thread[id=1255, name=Scheduler-1208377519, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
9) Thread[id=923, name=qtp66645842-923, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.EPoll.wait(Native Method)
at java.base@11/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:120)
at java.base@11/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:124)
at java.base@11/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
at app//org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at app//org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132)
at app//org.eclipse.jetty.io.ManagedSelector$$Lambda$151/0x00007fc384df3858.run(Unknown Source)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
10) Thread[id=1002, name=qtp1645118601-1002-acceptor-0@cbb0765-ServerConnector@4695d09b{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:41353}, state=RUNNABLE, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:533)
at java.base@11/sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:285)
at app//org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
at app//org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
at app//org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
at java.base@11/java.lang.Thread.run(Thread.java:834)
11) Thread[id=1229, name=SolrRrdBackendFactory-242-thread-1, state=TIMED_WAITING, group=TGRP-ChaosMonkeySafeLeaderTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1182)
at java.base@11/java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:899)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.Thread
[...truncated too long message...]
d-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 130197 INFO (closeThreadPool-306-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72167245759119366-127.0.0.1:45179_solr-n_0000000000) closing
[junit4] 2> 130197 INFO (OverseerStateUpdate-72167245759119366-127.0.0.1:45179_solr-n_0000000000) [n:127.0.0.1:45179_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:45179_solr
[junit4] 2> 130197 INFO (OverseerAutoScalingTriggerThread-72167245759119366-127.0.0.1:45179_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 130197 DEBUG (closeThreadPool-306-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Shutting down action executor now
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ] o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool executor
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ] o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
[junit4] 2> 130198 DEBUG (closeThreadPool-306-thread-1) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
[junit4] 2> 130201 INFO (closeThreadPool-306-thread-2) [ ] o.a.s.c.Overseer Overseer (id=72167245759119366-127.0.0.1:45179_solr-n_0000000000) closing
[junit4] 2> 130202 INFO (jetty-closer-253-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72167245759119366-127.0.0.1:45179_solr-n_0000000000) closing
[junit4] 2> 130203 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x10063ba47e40006, likely client has closed socket
[junit4] 2> 130203 INFO (jetty-closer-253-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@6e7b5b89{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:0}
[junit4] 2> 130203 INFO (jetty-closer-253-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@7694e57c{/solr,null,UNAVAILABLE}
[junit4] 2> 130203 INFO (jetty-closer-253-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 130204 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 130205 ERROR (SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 130205 WARN (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/collections/AutoscalingHistoryHandlerTest_collection/terms/shard1
[junit4] 2> 6 /solr/aliases.json
[junit4] 2> 3 /solr/collections/.system/terms/shard1
[junit4] 2> 3 /solr/collections/AutoscalingHistoryHandlerTest_collection/state.json
[junit4] 2> 2 /solr/security.json
[junit4] 2> 2 /solr/configs/AutoscalingHistoryHandlerTest_collection.AUTOCREATED/managed-schema
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 9 /solr/collections/.system/state.json
[junit4] 2> 6 /solr/clusterstate.json
[junit4] 2> 6 /solr/collections/AutoscalingHistoryHandlerTest_collection/state.json
[junit4] 2> 6 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 6 /solr/live_nodes
[junit4] 2> 6 /solr/collections
[junit4] 2>
[junit4] 2> 130205 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45849
[junit4] 2> 130205 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 45849
[junit4] 2> 130205 INFO (SUITE-AutoscalingHistoryHandlerTest-seed#[21BB66BA1FDDE245]-worker) [ ] o.a.s.SolrTestCaseJ4 ------------------------------------------------------- Done waiting for tracked resources to be released
[junit4] 2> Dec 10, 2018 2:27:14 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 9 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {}, docValues:{}, maxPointsInLeafNode=734, maxMBSortInHeap=5.710486773639191, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@378836c5), locale=ta-SG, timezone=SystemV/YST9YDT
[junit4] 2> NOTE: Linux 4.15.0-39-generic amd64/Oracle Corporation 11 (64-bit)/cpus=8,threads=1,free=123447192,total=512229376
[junit4] 2> NOTE: All tests run in this JVM: [ChaosMonkeySafeLeaderTest, AutoscalingHistoryHandlerTest, AutoscalingHistoryHandlerTest]
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=AutoscalingHistoryHandlerTest -Dtests.seed=21BB66BA1FDDE245 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=ta-SG -Dtests.timezone=SystemV/YST9YDT -Dtests.asserts=true -Dtests.file.encoding=UTF-8
[junit4] ERROR 0.00s J2 | AutoscalingHistoryHandlerTest (suite) <<<
[junit4] > Throwable #1: java.lang.RuntimeException: Failed while waiting for active collection
[junit4] > Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null
[junit4] > Live Nodes: [127.0.0.1:40289_solr, 127.0.0.1:45179_solr]
[junit4] > Last available state: null
[junit4] > at __randomizedtesting.SeedInfo.seed([21BB66BA1FDDE245]:0)
[junit4] > at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
[junit4] > at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
[junit4] > at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] Completed [9/10 (2!)] on J2 in 45.56s, 0 tests, 1 error <<< FAILURES!
[junit4]
[junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[junit4] OK 32.4s J0 | AutoscalingHistoryHandlerTest.testHistory
[junit4] Completed [10/10 (2!)] on J0 in 37.98s, 1 test
[junit4]
[junit4]
[junit4] Tests with failures [seed: 21BB66BA1FDDE245]:
[junit4] - org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest (suite)
[junit4] - org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest (suite)
[junit4]
[junit4]
[junit4] JVM J0: 0.71 .. 142.21 = 141.50s
[junit4] JVM J1: 0.71 .. 106.42 = 105.71s
[junit4] JVM J2: 0.70 .. 131.40 = 130.70s
[junit4] Execution time total: 2 minutes 22 seconds
[junit4] Tests summary: 10 suites, 8 tests, 2 suite-level errors
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-http2-Linux/lucene/common-build.xml:1572: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-http2-Linux/lucene/common-build.xml:1099: There were test failures: 10 suites, 8 tests, 2 suite-level errors [seed: 21BB66BA1FDDE245]
Total time: 2 minutes 23 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 0/5 failed: org.apache.solr.client.solrj.embedded.SolrExampleStreamingBinaryHttp2Test
[repro] 0/5 failed: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[repro] 2/5 failed: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[repro] 5/5 failed: org.apache.solr.client.solrj.io.stream.StreamingTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=0ecb8110eafb6b33b445f94c0bf171307b5a7a71, workspace=/var/lib/jenkins/workspace/Lucene-Solr-http2-Linux
[WARNINGS] Computing warning deltas based on reference build #55
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[JENKINS-EA] Lucene-Solr-http2-Linux (64bit/jdk-12-ea+12) - Build #
55 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/55/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseG1GC
13 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Failed while waiting for active collection Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null Live Nodes: [127.0.0.1:35421_solr, 127.0.0.1:43445_solr] Last available state: null
Stack Trace:
java.lang.RuntimeException: Failed while waiting for active collection
Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null
Live Nodes: [127.0.0.1:35421_solr, 127.0.0.1:43445_solr]
Last available state: null
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934]:0)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Failed while waiting for active collection Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null Live Nodes: [127.0.0.1:32911_solr, 127.0.0.1:34705_solr] Last available state: null
Stack Trace:
java.lang.RuntimeException: Failed while waiting for active collection
Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null
Live Nodes: [127.0.0.1:32911_solr, 127.0.0.1:34705_solr]
Last available state: null
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934]:0)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Failed while waiting for active collection Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null Live Nodes: [127.0.0.1:42109_solr, 127.0.0.1:46659_solr] Last available state: null
Stack Trace:
java.lang.RuntimeException: Failed while waiting for active collection
Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null
Live Nodes: [127.0.0.1:42109_solr, 127.0.0.1:46659_solr]
Last available state: null
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934]:0)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Failed while waiting for active collection Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null Live Nodes: [127.0.0.1:35217_solr, 127.0.0.1:40683_solr] Last available state: null
Stack Trace:
java.lang.RuntimeException: Failed while waiting for active collection
Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null
Live Nodes: [127.0.0.1:35217_solr, 127.0.0.1:40683_solr]
Last available state: null
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934]:0)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Failed while waiting for active collection Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null Live Nodes: [127.0.0.1:35015_solr, 127.0.0.1:40811_solr] Last available state: null
Stack Trace:
java.lang.RuntimeException: Failed while waiting for active collection
Timeout waiting to see state for collection=AutoscalingHistoryHandlerTest_collection :null
Live Nodes: [127.0.0.1:35015_solr, 127.0.0.1:40811_solr]
Last available state: null
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934]:0)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:736)
at org.apache.solr.cloud.MiniSolrCloudCluster.waitForActiveCollection(MiniSolrCloudCluster.java:742)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.SplitShardTest.doTest
Error Message:
Error from server at https://127.0.0.1:37949/solr: SPLITSHARD timed out waiting for subshard leaders to come up
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:37949/solr: SPLITSHARD timed out waiting for subshard leaders to come up
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:13AA413FB373FA8D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:196)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at org.apache.solr.cloud.SplitShardTest.doTest(SplitShardTest.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.SplitShardTest.doTest
Error Message:
Error from server at https://127.0.0.1:42117/solr: SPLITSHARD timed out waiting for subshard leaders to come up
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:42117/solr: SPLITSHARD timed out waiting for subshard leaders to come up
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:13AA413FB373FA8D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:196)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at org.apache.solr.cloud.SplitShardTest.doTest(SplitShardTest.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.SplitShardTest.doTest
Error Message:
Error from server at https://127.0.0.1:37995/solr: SPLITSHARD timed out waiting for subshard leaders to come up
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:37995/solr: SPLITSHARD timed out waiting for subshard leaders to come up
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:13AA413FB373FA8D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:196)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at org.apache.solr.cloud.SplitShardTest.doTest(SplitShardTest.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.SplitShardTest.doTest
Error Message:
Error from server at https://127.0.0.1:35617/solr: SPLITSHARD timed out waiting for subshard leaders to come up
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:35617/solr: SPLITSHARD timed out waiting for subshard leaders to come up
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:13AA413FB373FA8D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:196)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at org.apache.solr.cloud.SplitShardTest.doTest(SplitShardTest.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.SplitShardTest.doTest
Error Message:
Error from server at https://127.0.0.1:41335/solr: SPLITSHARD timed out waiting for subshard leaders to come up
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:41335/solr: SPLITSHARD timed out waiting for subshard leaders to come up
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:13AA413FB373FA8D]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:196)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at org.apache.solr.cloud.SplitShardTest.doTest(SplitShardTest.java:67)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition
Error Message:
expected:<9> but was:<6>
Stack Trace:
java.lang.AssertionError: expected:<9> but was:<6>
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:E1639476B2A6109F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition(CollectionsAPIAsyncDistributedZkTest.java:274)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition
Error Message:
expected:<9> but was:<8>
Stack Trace:
java.lang.AssertionError: expected:<9> but was:<8>
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:E1639476B2A6109F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition(CollectionsAPIAsyncDistributedZkTest.java:274)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
FAILED: org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition
Error Message:
expected:<9> but was:<8>
Stack Trace:
java.lang.AssertionError: expected:<9> but was:<8>
at __randomizedtesting.SeedInfo.seed([B4EEF99BDEC8E934:E1639476B2A6109F]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest.testAsyncIdRaceCondition(CollectionsAPIAsyncDistributedZkTest.java:274)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)
Build Log:
[...truncated 13560 lines...]
[junit4] Suite: org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_B4EEF99BDEC8E934-001/init-core-data-001
[junit4] 2> 489265 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_B4EEF99BDEC8E934-001/tempDir-001
[junit4] 2> 489265 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 489266 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 489266 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 489267 ERROR (ZkTestServer Run Thread) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 489366 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer start zk server on port:36067
[junit4] 2> 489366 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:36067
[junit4] 2> 489366 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 36067
[junit4] 2> 489368 INFO (zkConnectionManagerCallback-3511-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489369 INFO (zkConnectionManagerCallback-3513-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489370 INFO (zkConnectionManagerCallback-3515-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489373 WARN (jetty-launcher-3508-thread-1) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 489373 WARN (jetty-launcher-3508-thread-2) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 489374 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 489374 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 489374 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 489374 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 489374 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 12-ea+12
[junit4] 2> 489374 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 12-ea+12
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@392b33aa{/solr,null,AVAILABLE}
[junit4] 2> 489375 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60fcd656{/solr,null,AVAILABLE}
[junit4] 2> 489377 WARN (jetty-launcher-3508-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@133e9e2e[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489377 WARN (jetty-launcher-3508-thread-2) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@509559f8[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@61719893{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:33241}
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@54d26e5c{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:35805}
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.e.j.s.Server Started @489417ms
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.e.j.s.Server Started @489417ms
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=33241}
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=35805}
[junit4] 2> 489377 ERROR (jetty-launcher-3508-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 489377 ERROR (jetty-launcher-3508-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-12-09T20:46:18.012938Z
[junit4] 2> 489377 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-12-09T20:46:18.012895Z
[junit4] 2> 489379 INFO (zkConnectionManagerCallback-3519-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489379 INFO (zkConnectionManagerCallback-3518-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489379 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 489379 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 489455 WARN (jetty-launcher-3508-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@c1a1290[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489459 WARN (jetty-launcher-3508-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@19714397[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489460 INFO (jetty-launcher-3508-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36067/solr
[junit4] 2> 489461 INFO (zkConnectionManagerCallback-3523-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489464 INFO (zkConnectionManagerCallback-3525-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489478 WARN (jetty-launcher-3508-thread-2) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@45613984[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489480 WARN (jetty-launcher-3508-thread-2) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@7997d1b9[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489481 INFO (jetty-launcher-3508-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:36067/solr
[junit4] 2> 489481 INFO (zkConnectionManagerCallback-3531-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489483 INFO (zkConnectionManagerCallback-3533-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489486 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:35805_solr
[junit4] 2> 489486 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.c.Overseer Overseer (id=72165908088225800-127.0.0.1:35805_solr-n_0000000000) starting
[junit4] 2> 489488 INFO (zkConnectionManagerCallback-3540-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489489 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36067/solr ready
[junit4] 2> 489489 INFO (OverseerStateUpdate-72165908088225800-127.0.0.1:35805_solr-n_0000000000) [n:127.0.0.1:35805_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:35805_solr
[junit4] 2> 489490 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35805_solr
[junit4] 2> 489490 INFO (zkCallback-3532-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 489490 INFO (zkCallback-3539-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 489491 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 489491 INFO (OverseerAutoScalingTriggerThread-72165908088225800-127.0.0.1:35805_solr-n_0000000000) [ ] o.a.s.c.a.NodeLostTrigger NodeLostTrigger .auto_add_replicas - Initial livenodes: [127.0.0.1:35805_solr]
[junit4] 2> 489492 INFO (ScheduledTrigger-3533-thread-1) [ ] o.a.s.c.a.NodeLostTrigger Running NodeLostTrigger: .auto_add_replicas with currently live nodes: 1 and last live nodes: 1
[junit4] 2> 489492 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.ZkController Publish node=127.0.0.1:33241_solr as DOWN
[junit4] 2> 489492 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 489492 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33241_solr
[junit4] 2> 489493 INFO (zkCallback-3539-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 489493 INFO (zkCallback-3532-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 489493 INFO (zkCallback-3524-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 489502 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 489504 INFO (zkConnectionManagerCallback-3546-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489505 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 489506 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36067/solr ready
[junit4] 2> 489506 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 489514 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35805.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73fc4a39
[junit4] 2> 489515 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33241.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73fc4a39
[junit4] 2> 489519 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35805.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73fc4a39
[junit4] 2> 489520 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_35805.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73fc4a39
[junit4] 2> 489520 INFO (jetty-launcher-3508-thread-2) [n:127.0.0.1:35805_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_B4EEF99BDEC8E934-001/tempDir-001/node2/.
[junit4] 2> 489520 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33241.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73fc4a39
[junit4] 2> 489521 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_33241.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@73fc4a39
[junit4] 2> 489521 INFO (jetty-launcher-3508-thread-1) [n:127.0.0.1:33241_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_B4EEF99BDEC8E934-001/tempDir-001/node1/.
[junit4] 2> 489554 INFO (zkConnectionManagerCallback-3552-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489554 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 489555 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:36067/solr ready
[junit4] 2> 489579 INFO (qtp498779045-15823) [n:127.0.0.1:35805_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params async=testAsyncIdBackCompat&collection.configName=conf1&name=testAsyncIdBackCompat&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 489580 INFO (qtp498779045-15823) [n:127.0.0.1:35805_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={async=testAsyncIdBackCompat&collection.configName=conf1&name=testAsyncIdBackCompat&nrtReplicas=1&action=CREATE&numShards=1&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 489582 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1351767308
[junit4] 2> 489582 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:35805_solr
[junit4] 2> 489582 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.c.CoreContainer Publish this node as DOWN...
[junit4] 2> 489582 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.c.ZkController Publish node=127.0.0.1:35805_solr as DOWN
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1963452349
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:33241_solr
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.c.CoreContainer Publish this node as DOWN...
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.c.ZkController Publish node=127.0.0.1:33241_solr as DOWN
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 489583 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2398d5d8: rootName = solr_35805, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4c808d2d
[junit4] 2> 489584 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 489584 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@38cad140: rootName = solr_33241, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@35f804a
[junit4] 2> 489587 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 489587 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@b5c042e: rootName = solr_33241, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@3190d217
[junit4] 2> 489588 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 489588 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@183ab042: rootName = solr_35805, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@3190d217
[junit4] 2> 489589 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 489589 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@329f941f: rootName = solr_33241, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4d055aac
[junit4] 2> 489589 INFO (jetty-closer-3509-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 489590 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 489590 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@31cfaae9: rootName = solr_35805, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@4d055aac
[junit4] 2> 489590 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 489594 INFO (closeThreadPool-3558-thread-2) [ ] o.a.s.c.Overseer Overseer (id=72165908088225800-127.0.0.1:35805_solr-n_0000000000) closing
[junit4] 2> 489594 INFO (OverseerAutoScalingTriggerThread-72165908088225800-127.0.0.1:35805_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 489595 INFO (OverseerStateUpdate-72165908088225800-127.0.0.1:35805_solr-n_0000000000) [n:127.0.0.1:35805_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:35805_solr
[junit4] 2> 489595 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006282d47e0009, likely client has closed socket
[junit4] 2> 489595 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006282d47e000a, likely client has closed socket
[junit4] 2> 489596 INFO (jetty-closer-3509-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72165908088225800-127.0.0.1:35805_solr-n_0000000000) closing
[junit4] 2> 489596 INFO (closeThreadPool-3558-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72165908088225800-127.0.0.1:35805_solr-n_0000000000) closing
[junit4] 2> 489596 INFO (jetty-closer-3509-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@61719893{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:0}
[junit4] 2> 489596 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006282d47e0006, likely client has closed socket
[junit4] 2> 489596 INFO (jetty-closer-3509-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@60fcd656{/solr,null,UNAVAILABLE}
[junit4] 2> 489596 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006282d47e0008, likely client has closed socket
[junit4] 2> 489596 INFO (jetty-closer-3509-thread-2) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 489597 INFO (jetty-closer-3509-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@54d26e5c{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:0}
[junit4] 2> 489597 INFO (jetty-closer-3509-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@392b33aa{/solr,null,UNAVAILABLE}
[junit4] 2> 489597 INFO (jetty-closer-3509-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 489598 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 489598 ERROR (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 489598 WARN (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 2 /solr/security.json
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> 489599 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:36067
[junit4] 2> 489599 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncIdBackCompat-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 36067
[junit4] 2> 489603 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncRequests-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_B4EEF99BDEC8E934-001/tempDir-002
[junit4] 2> 489603 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncRequests-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 489603 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 489603 INFO (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 489604 ERROR (ZkTestServer Run Thread) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 489703 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncRequests-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer start zk server on port:37209
[junit4] 2> 489703 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncRequests-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:37209
[junit4] 2> 489703 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testAsyncRequests-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 37209
[junit4] 2> 489705 INFO (zkConnectionManagerCallback-3562-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489707 INFO (zkConnectionManagerCallback-3564-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489708 INFO (zkConnectionManagerCallback-3566-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489711 WARN (jetty-launcher-3559-thread-1) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 489711 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 489711 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 489711 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 12-ea+12
[junit4] 2> 489712 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 489712 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 489712 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 489712 WARN (jetty-launcher-3559-thread-2) [ ] o.e.j.s.AbstractConnector Ignoring deprecated socket close linger time
[junit4] 2> 489712 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@39802a4b{/solr,null,AVAILABLE}
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Start Jetty (original configured port=0)
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.Server jetty-9.4.14.v20181114; built: 2018-11-14T21:20:31.478Z; git: c4550056e785fb5665914545889f21dc136ad9e6; jvm 12-ea+12
[junit4] 2> 489713 WARN (jetty-launcher-3559-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@2a17f895[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@15547c9{/solr,null,AVAILABLE}
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@5e6f075{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:34999}
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.e.j.s.Server Started @489753ms
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=34999}
[junit4] 2> 489713 ERROR (jetty-launcher-3559-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 489713 WARN (jetty-launcher-3559-thread-2) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@44d73f18[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 489713 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-12-09T20:46:18.348893Z
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@45af9f68{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:41037}
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.e.j.s.Server Started @489753ms
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41037}
[junit4] 2> 489714 ERROR (jetty-launcher-3559-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr? version 8.0.0
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 489714 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-12-09T20:46:18.349316Z
[junit4] 2> 489719 INFO (zkConnectionManagerCallback-3568-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489719 INFO (zkConnectionManagerCallback-3570-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489720 INFO (jetty-launcher-3559-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 489720 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 489723 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x1006282d5cf0004, likely client has closed socket
[junit4] 2> 489783 WARN (jetty-launcher-3559-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@fa8b417[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489796 WARN (jetty-launcher-3559-thread-2) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@58aab53a[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489796 WARN (jetty-launcher-3559-thread-1) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@2a820f0e[provider=null,keyStore=null,trustStore=null]
[junit4] 2> 489797 INFO (jetty-launcher-3559-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:37209/solr
[junit4] 2> 489798 INFO (zkConnectionManagerCallback-3575-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 489799 INFO (zkConnectionManagerCallback-3577-
[...truncated too long message...]
2> 183715 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 183715 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@432fa1f: rootName = solr_44887, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@4ad71b24
[junit4] 2> 183719 INFO (jetty-closer-634-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 183719 INFO (jetty-closer-634-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@381aa594: rootName = solr_40233, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@38257c4d
[junit4] 2> 183719 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 183719 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6562b3b2: rootName = solr_44887, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@38257c4d
[junit4] 2> 183721 INFO (jetty-closer-634-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 183722 INFO (jetty-closer-634-thread-2) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2952c10b: rootName = solr_40233, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@421c4eb1
[junit4] 2> 183722 INFO (jetty-closer-634-thread-2) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 183722 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 183722 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@725e11b8: rootName = solr_44887, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@421c4eb1
[junit4] 2> 183722 INFO (jetty-closer-634-thread-1) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 183724 INFO (closeThreadPool-686-thread-2) [ ] o.a.s.c.Overseer Overseer (id=72166031984558086-127.0.0.1:44887_solr-n_0000000000) closing
[junit4] 2> 183724 INFO (OverseerStateUpdate-72166031984558086-127.0.0.1:44887_solr-n_0000000000) [n:127.0.0.1:44887_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:44887_solr
[junit4] 2> 183725 INFO (OverseerAutoScalingTriggerThread-72166031984558086-127.0.0.1:44887_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 183726 INFO (zkCallback-664-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 183726 INFO (zkCallback-669-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (1)
[junit4] 2> 183727 INFO (zkCallback-664-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 183732 INFO (closeThreadPool-686-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72166031984558086-127.0.0.1:44887_solr-n_0000000000) closing
[junit4] 2> 183732 INFO (jetty-closer-634-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72166031984558086-127.0.0.1:44887_solr-n_0000000000) closing
[junit4] 2> 183735 INFO (zkCallback-669-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (0)
[junit4] 2> 183738 INFO (jetty-closer-634-thread-1) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@afb55c4{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:0}
[junit4] 2> 183738 INFO (jetty-closer-634-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@560cd3f7{/solr,null,UNAVAILABLE}
[junit4] 2> 183738 INFO (jetty-closer-634-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 183738 INFO (jetty-closer-634-thread-2) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@a2be743{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:0}
[junit4] 2> 183738 INFO (jetty-closer-634-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@4fea10be{/solr,null,UNAVAILABLE}
[junit4] 2> 183738 INFO (jetty-closer-634-thread-2) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 183739 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer Shutting down ZkTestServer.
[junit4] 2> 183739 ERROR (TEST-CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls-seed#[B4EEF99BDEC8E934]) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 183739 WARN (ZkTestServer Run Thread) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 10 /solr/collections/testasynccollectioncreation/terms/shard1_1
[junit4] 2> 10 /solr/collections/testasynccollectioncreation/terms/shard1_0
[junit4] 2> 5 /solr/aliases.json
[junit4] 2> 5 /solr/collections/testasynccollectioncreation/terms/shard1
[junit4] 2> 2 /solr/security.json
[junit4] 2> 2 /solr/configs/conf1
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 87 /solr/collections/testasynccollectioncreation/state.json
[junit4] 2> 5 /solr/clusterstate.json
[junit4] 2> 5 /solr/clusterprops.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 5 /solr/live_nodes
[junit4] 2> 5 /solr/collections
[junit4] 2>
[junit4] 2> 183740 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:37981
[junit4] 2> 183740 INFO (TEST-CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls-seed#[B4EEF99BDEC8E934]) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 37981
[junit4] OK 14.5s J2 | CollectionsAPIAsyncDistributedZkTest.testSolrJAPICalls
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-http2-Linux/solr/build/solr-core/test/J2/temp/solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest_B4EEF99BDEC8E934-002
[junit4] 2> Dec 09, 2018 9:18:02 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
[junit4] 2> WARNING: Will linger awaiting termination of 8 leaked thread(s).
[junit4] 2> NOTE: test params are: codec=Lucene80, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@cf464f3), locale=no-NO, timezone=Africa/Kinshasa
[junit4] 2> NOTE: Linux 4.15.0-39-generic amd64/Oracle Corporation 12-ea (64-bit)/cpus=8,threads=1,free=226960328,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [CollectionsAPIAsyncDistributedZkTest, CollectionsAPIAsyncDistributedZkTest, AutoscalingHistoryHandlerTest, CollectionsAPIAsyncDistributedZkTest]
[junit4] Completed [15/15 (10!)] on J2 in 44.13s, 4 tests, 1 failure <<< FAILURES!
[...truncated 20 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-http2-Linux/lucene/common-build.xml:1572: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-http2-Linux/lucene/common-build.xml:1099: There were test failures: 15 suites, 31 tests, 4 suite-level errors, 4 errors, 2 failures [seed: B4EEF99BDEC8E934]
Total time: 3 minutes 6 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 2/5 failed: org.apache.solr.cloud.api.collections.CollectionsAPIAsyncDistributedZkTest
[repro] 4/5 failed: org.apache.solr.cloud.SplitShardTest
[repro] 4/5 failed: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=0ecb8110eafb6b33b445f94c0bf171307b5a7a71, workspace=/var/lib/jenkins/workspace/Lucene-Solr-http2-Linux
[WARNINGS] Computing warning deltas based on reference build #54
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2