You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/09/12 20:00:40 UTC
[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build #
2060 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/2060/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC
3 tests failed.
FAILED: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest.test
Error Message:
Error from server at http://127.0.0.1:33247: At least one of the node(s) specified [127.0.0.1:49911_] are not currently active in [], no action taken.
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:33247: At least one of the node(s) specified [127.0.0.1:49911_] are not currently active in [], no action taken.
at __randomizedtesting.SeedInfo.seed([D9B83986EA2BDE2C:51EC065C44D7B3D4]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:425)
at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1006)
at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: junit.framework.TestSuite.org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
Error Message:
87 threads leaked from SUITE scope at org.apache.solr.metrics.reporters.solr.SolrShardReporterTest: 1) Thread[id=8073, name=qtp1894331194-8073, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 2) Thread[id=8099, name=zkConnectionManagerCallback-2270-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 3) Thread[id=8106, name=Scheduler-1963924670, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) Thread[id=8188, name=SolrRrdBackendFactory-3782-thread-2, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 5) Thread[id=8037, name=qtp1681156577-8037, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 6) Thread[id=8092, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 7) Thread[id=8186, name=ScheduledTrigger-3788-thread-4, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 8) Thread[id=8153, name=OverseerAutoScalingTriggerThread-72325171990953994-127.0.0.1:55924_-n_0000000001, state=WAITING, group=Overseer autoscaling triggers] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189) at java.lang.Thread.run(Thread.java:748) 9) Thread[id=8121, name=Scheduler-1558059189, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 10) Thread[id=8036, name=qtp1681156577-8036, state=RUNNABLE, group=TGRP-SolrShardReporterTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 11) Thread[id=8185, name=SolrRrdBackendFactory-3754-thread-2, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 12) Thread[id=8061, name=MetricsHistoryHandler-3755-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 13) Thread[id=8063, name=Thread-1407, state=WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1857) 14) Thread[id=8079, name=qtp1894331194-8079, state=RUNNABLE, group=TGRP-SolrShardReporterTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 15) Thread[id=8119, name=qtp1492061395-8119, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 16) Thread[id=8135, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 17) Thread[id=8103, name=Thread-1416, state=WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1857) 18) Thread[id=8118, name=qtp1492061395-8118, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 19) Thread[id=8051, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 20) Thread[id=8114, name=qtp1492061395-8114-acceptor-0@36c1dd31-ServerConnector@5fb736bc{HTTP/1.1,[http/1.1]}{127.0.0.1:49911}, state=RUNNABLE, group=TGRP-SolrShardReporterTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 21) Thread[id=8087, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 22) Thread[id=8134, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 23) Thread[id=8112, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 24) Thread[id=8120, name=qtp1492061395-8120, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 25) Thread[id=8074, name=qtp1894331194-8074-acceptor-0@63e657d8-ServerConnector@1d42d86e{HTTP/1.1,[http/1.1]}{127.0.0.1:57698}, state=RUNNABLE, group=TGRP-SolrShardReporterTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 26) Thread[id=8102, name=metrics-solr-reporter-12-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 27) Thread[id=8187, name=SolrRrdBackendFactory-3768-thread-2, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 28) Thread[id=8098, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 29) Thread[id=8126, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 30) Thread[id=8078, name=qtp1894331194-8078, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 31) Thread[id=8072, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 32) Thread[id=8131, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 33) Thread[id=8142, name=metrics-solr-reporter-14-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 34) Thread[id=8085, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 35) Thread[id=8045, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 36) Thread[id=8125, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 37) Thread[id=8075, name=qtp1894331194-8075, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 38) Thread[id=8183, name=ScheduledTrigger-3788-thread-2, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:317) at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:58) at org.apache.solr.cloud.autoscaling.TriggerBase.restoreState(TriggerBase.java:252) at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:583) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 39) Thread[id=8077, name=qtp1894331194-8077, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 40) Thread[id=8033, name=qtp1681156577-8033, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 41) Thread[id=8060, name=SolrRrdBackendFactory-3754-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 42) Thread[id=8136, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 43) Thread[id=8040, name=qtp1681156577-8040, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 44) Thread[id=8053, name=zkConnectionManagerCallback-2247-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 45) Thread[id=8056, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 46) Thread[id=8107, name=metrics-solr-reporter-13-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 47) Thread[id=8154, name=ScheduledTrigger-3788-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 48) Thread[id=8093, name=zkConnectionManagerCallback-2263-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 49) Thread[id=8109, name=searcherExecutor-3771-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 50) Thread[id=8115, name=qtp1492061395-8115, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 51) Thread[id=8034, name=qtp1681156577-8034-acceptor-0@31680632-ServerConnector@66ce9cb2{HTTP/1.1,[http/1.1]}{127.0.0.1:55924}, state=RUNNABLE, group=TGRP-SolrShardReporterTest] at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369) at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 52) Thread[id=8038, name=qtp1681156577-8038, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 53) Thread[id=8052, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 54) Thread[id=8046, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 55) Thread[id=8096, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 56) Thread[id=8101, name=MetricsHistoryHandler-3769-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 57) Thread[id=8080, name=qtp1894331194-8080, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 58) Thread[id=8047, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 59) Thread[id=8152, name=OverseerCollectionConfigSetProcessor-72325171990953994-127.0.0.1:55924_-n_0000000001, state=TIMED_WAITING, group=Overseer collection creation process.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:341) at org.apache.solr.cloud.OverseerTaskProcessor.amILeader(OverseerTaskProcessor.java:398) at org.apache.solr.cloud.OverseerTaskProcessor.run(OverseerTaskProcessor.java:198) at java.lang.Thread.run(Thread.java:748) 60) Thread[id=8076, name=qtp1894331194-8076, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 61) Thread[id=8116, name=qtp1492061395-8116, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 62) Thread[id=8086, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 63) Thread[id=8141, name=MetricsHistoryHandler-3783-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 64) Thread[id=8091, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 65) Thread[id=8100, name=SolrRrdBackendFactory-3768-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 66) Thread[id=8133, name=zkConnectionManagerCallback-2279-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 67) Thread[id=8067, name=metrics-solr-reporter-11-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 68) Thread[id=8097, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 69) Thread[id=8058, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 70) Thread[id=8143, name=Thread-1425, state=WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at org.apache.solr.core.CloserThread.run(CoreContainer.java:1857) 71) Thread[id=8140, name=SolrRrdBackendFactory-3782-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 72) Thread[id=8035, name=qtp1681156577-8035, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) 73) Thread[id=8069, name=searcherExecutor-3757-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 74) Thread[id=8117, name=qtp1492061395-8117, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 75) Thread[id=8066, name=Scheduler-864862186, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 76) Thread[id=8059, name=zkConnectionManagerCallback-2254-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 77) Thread[id=8062, name=metrics-solr-reporter-10-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 78) Thread[id=8139, name=zkConnectionManagerCallback-2286-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 79) Thread[id=8041, name=Scheduler-1686928871, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 80) Thread[id=8127, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.lang.Thread.run(Thread.java:748) 81) Thread[id=8184, name=ScheduledTrigger-3788-thread-3, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.exists(SolrZkClient.java:317) at org.apache.solr.client.solrj.impl.ZkDistribStateManager.hasData(ZkDistribStateManager.java:58) at org.apache.solr.cloud.autoscaling.TriggerBase.saveState(TriggerBase.java:234) at org.apache.solr.cloud.autoscaling.ScheduledTriggers$TriggerWrapper.run(ScheduledTriggers.java:598) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 82) Thread[id=8151, name=OverseerStateUpdate-72325171990953994-127.0.0.1:55924_-n_0000000001, state=TIMED_WAITING, group=Overseer state updater.] at java.lang.Thread.sleep(Native Method) at org.apache.solr.common.cloud.ZkCmdExecutor.retryDelay(ZkCmdExecutor.java:111) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:70) at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:341) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.amILeader(Overseer.java:421) at org.apache.solr.cloud.Overseer$ClusterStateUpdater.run(Overseer.java:147) at java.lang.Thread.run(Thread.java:748) 83) Thread[id=8081, name=Scheduler-1235499399, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 84) Thread[id=8039, name=qtp1681156577-8039, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392) at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653) at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717) at java.lang.Thread.run(Thread.java:748) 85) Thread[id=8132, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 86) Thread[id=8057, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest] at java.lang.Thread.sleep(Native Method) at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 87) Thread[id=8113, name=qtp1492061395-8113, state=RUNNABLE, group=TGRP-SolrShardReporterTest] at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method) at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223) at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396) at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:132) at org.eclipse.jetty.io.ManagedSelector$$Lambda$28/1037677505.run(Unknown Source) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 87 threads leaked from SUITE scope at org.apache.solr.metrics.reporters.solr.SolrShardReporterTest:
1) Thread[id=8073, name=qtp1894331194-8073, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)
2) Thread[id=8099, name=zkConnectionManagerCallback-2270-thread-1, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
3) Thread[id=8106, name=Scheduler-1963924670, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
4) Thread[id=8188, name=SolrRrdBackendFactory-3782-thread-2, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
5) Thread[id=8037, name=qtp1681156577-8037, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
6) Thread[id=8092, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
7) Thread[id=8186, name=ScheduledTrigger-3788-thread-4, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
8) Thread[id=8153, name=OverseerAutoScalingTriggerThread-72325171990953994-127.0.0.1:55924_-n_0000000001, state=WAITING, group=Overseer autoscaling triggers]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:189)
at java.lang.Thread.run(Thread.java:748)
9) Thread[id=8121, name=Scheduler-1558059189, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
10) Thread[id=8036, name=qtp1681156577-8036, state=RUNNABLE, group=TGRP-SolrShardReporterTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
11) Thread[id=8185, name=SolrRrdBackendFactory-3754-thread-2, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
12) Thread[id=8061, name=MetricsHistoryHandler-3755-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
13) Thread[id=8063, name=Thread-1407, state=WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1857)
14) Thread[id=8079, name=qtp1894331194-8079, state=RUNNABLE, group=TGRP-SolrShardReporterTest]
at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:396)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:333)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
15) Thread[id=8119, name=qtp1492061395-8119, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)
16) Thread[id=8135, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
17) Thread[id=8103, name=Thread-1416, state=WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.apache.solr.core.CloserThread.run(CoreContainer.java:1857)
18) Thread[id=8118, name=qtp1492061395-8118, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)
19) Thread[id=8051, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-SendThread(127.0.0.1:38027), state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
20) Thread[id=8114, name=qtp1492061395-8114-acceptor-0@36c1dd31-ServerConnector@5fb736bc{HTTP/1.1,[http/1.1]}{127.0.0.1:49911}, state=RUNNABLE, group=TGRP-SolrShardReporterTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
21) Thread[id=8087, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
22) Thread[id=8134, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
23) Thread[id=8112, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
24) Thread[id=8120, name=qtp1492061395-8120, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)
25) Thread[id=8074, name=qtp1894331194-8074-acceptor-0@63e657d8-ServerConnector@1d42d86e{HTTP/1.1,[http/1.1]}{127.0.0.1:57698}, state=RUNNABLE, group=TGRP-SolrShardReporterTest]
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:369)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:639)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
at java.lang.Thread.run(Thread.java:748)
26) Thread[id=8102, name=metrics-solr-reporter-12-thread-1, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
27) Thread[id=8187, name=SolrRrdBackendFactory-3768-thread-2, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
28) Thread[id=8098, name=TEST-SolrShardReporterTest.test-seed#[D9B83986EA2BDE2C]-EventThread, state=WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
29) Thread[id=8126, name=Connection evictor, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at java.lang.Thread.sleep(Native Method)
at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.lang.Thread.run(Thread.java:748)
30) Thread[id=8078, name=qtp1894331194-8078, state=TIMED_WAITING, group=TGRP-SolrShardReporterTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:653)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:48)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:717)
at java.lang.Thread.run(Thread.java:748)
31) Thread[id=8072
[...truncated too long message...]
d or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] Creating empty /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test/org/apache/solr/cloud/autoscaling/sim/package-info.class
[copy] Copying 1 file to /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test
common-solr.compile-test:
compile-test:
BUILD SUCCESSFUL
Total time: 59 seconds
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.SolrShardReporterTest" -Dtests.showOutput=onerror "-Dargs=-XX:+UseCompressedOops -XX:+UseG1GC" -Dtests.seed=D9B83986EA2BDE2C -Dtests.slow=true -Dtests.locale=en-CA -Dtests.timezone=America/La_Paz -Dtests.asserts=true -Dtests.file.encoding=UTF-8
Buildfile: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/build.xml
-clover.disable:
ivy-configure:
[ivy:configure] :: Apache Ivy 2.4.0 - 20141213170938 :: http://ant.apache.org/ivy/ ::
[ivy:configure] :: loading settings :: file = /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/top-level-ivy-settings.xml
install-junit4-taskdef:
ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.
-ivy-fail-disallowed-ivy-version:
ivy-fail:
resolve-groovy:
[ivy:cachepath] :: resolving dependencies :: org.codehaus.groovy#groovy-all-caller;working
[ivy:cachepath] confs: [default]
[ivy:cachepath] found org.codehaus.groovy#groovy-all;2.4.15 in public
[ivy:cachepath] :: resolution report :: resolve 21ms :: artifacts dl 1ms
---------------------------------------------------------------------
| | modules || artifacts |
| conf | number| search|dwnlded|evicted|| number|dwnlded|
---------------------------------------------------------------------
| default | 1 | 0 | 0 | 0 || 1 | 0 |
---------------------------------------------------------------------
-init-totals:
-test:
[mkdir] Created dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test
[junit4:pickseed] Seed property 'tests.seed' already defined: D9B83986EA2BDE2C
[mkdir] Created dir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/temp
[junit4] <JUnit4> says hello! Master seed: D9B83986EA2BDE2C
[junit4] Executing 5 suites with 2 JVMs.
[junit4]
[junit4] Started J1 PID(13082@serv1-vm3).
[junit4] Started J0 PID(13083@serv1-vm3).
[junit4] Suite: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
[junit4] OK 34.4s J0 | SolrShardReporterTest.test
[junit4] Completed [1/5] on J0 in 38.10s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
[junit4] OK 35.4s J1 | SolrShardReporterTest.test
[junit4] Completed [2/5] on J1 in 38.73s, 1 test
[junit4]
[junit4] Duplicate suite name used with XML reports: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest. This may confuse tools that process XML reports. Set 'ignoreDuplicateSuites' to true to skip this message.
[junit4] Suite: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
[junit4] OK 30.4s J0 | SolrShardReporterTest.test
[junit4] Completed [3/5] on J0 in 30.68s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
[junit4] OK 30.9s J1 | SolrShardReporterTest.test
[junit4] Completed [4/5] on J1 in 31.73s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
[junit4] OK 29.0s J0 | SolrShardReporterTest.test
[junit4] Completed [5/5] on J0 in 29.16s, 1 test
[junit4]
[junit4] JVM J0: 1.10 .. 100.39 = 99.29s
[junit4] JVM J1: 1.09 .. 73.20 = 72.11s
[junit4] Execution time total: 1 minute 40 seconds
[junit4] Tests summary: 5 suites, 5 tests
[junit4] Could not remove temporary path: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp])
[junit4] Could not remove temporary path: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0 (java.nio.file.DirectoryNotEmptyException: Remaining files: [/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp])
[echo] 5 slowest tests:
[junit4:tophints] 113.10s | org.apache.solr.cloud.ZkFailoverTest
[junit4:tophints] 99.52s | org.apache.solr.cloud.TestPullReplica
[junit4:tophints] 84.80s | org.apache.solr.cloud.TestWithCollection
[junit4:tophints] 70.36s | org.apache.solr.cloud.PeerSyncReplicationTest
[junit4:tophints] 67.16s | org.apache.solr.cloud.BasicDistributedZkTest
-check-totals:
test-nocompile:
BUILD SUCCESSFUL
Total time: 1 minute 43 seconds
[repro] Failures:
[repro] 0/5 failed: org.apache.solr.metrics.reporters.solr.SolrShardReporterTest
[repro] Exiting with code 0
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
java.lang.InterruptedException: no matches found within 10000
at hudson.FilePath$ValidateAntFileMask.hasMatch(FilePath.java:2845)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2724)
at hudson.FilePath$ValidateAntFileMask.invoke(FilePath.java:2705)
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3084)
Also: hudson.remoting.Channel$CallSiteStackTrace: Remote call to Solaris VBOX
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
at hudson.remoting.Channel.call(Channel.java:955)
at hudson.FilePath.act(FilePath.java:1070)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1840)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
Caused: hudson.FilePath$TunneledInterruptedException
at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3086)
at hudson.remoting.UserRequest.perform(UserRequest.java:212)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request$2.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused: java.lang.InterruptedException: java.lang.InterruptedException: no matches found within 10000
at hudson.FilePath.act(FilePath.java:1072)
at hudson.FilePath.act(FilePath.java:1059)
at hudson.FilePath.validateAntFileMask(FilePath.java:2703)
at hudson.tasks.ArtifactArchiver.perform(ArtifactArchiver.java:243)
at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
at hudson.model.Build$BuildExecution.post2(Build.java:186)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
at hudson.model.Run.execute(Run.java:1840)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:429)
No artifacts found that match the file pattern "**/*.events,heapdumps/**,**/*_pid*.log". Configuration error?
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=5b96f89d2b038bff2ed3351887a87108f7cc6ea3, workspace=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris
[WARNINGS] Computing warning deltas based on reference build #2059
Recording test results
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build #
2061 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/2061/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC
5 tests failed.
FAILED: org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
Error Message:
Error from server at http://127.0.0.1:39913/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39913/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
at __randomizedtesting.SeedInfo.seed([BAD8350446A6B372:34D4557433BE3758]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
Error Message:
Error from server at http://127.0.0.1:63432/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:63432/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
at __randomizedtesting.SeedInfo.seed([BAD8350446A6B372:34D4557433BE3758]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
Error Message:
Error from server at http://127.0.0.1:36071/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:36071/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
at __randomizedtesting.SeedInfo.seed([BAD8350446A6B372:34D4557433BE3758]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
Error Message:
Error from server at http://127.0.0.1:39484/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:39484/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
at __randomizedtesting.SeedInfo.seed([BAD8350446A6B372:34D4557433BE3758]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
Error Message:
Error from server at http://127.0.0.1:51108/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:51108/solr: Could not find collection : pull_replica_error_handling_test_cant_connect_to_leader
at __randomizedtesting.SeedInfo.seed([BAD8350446A6B372:34D4557433BE3758]:0)
at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.cloud.TestPullReplicaErrorHandling.tearDown(TestPullReplicaErrorHandling.java:128)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 12794 lines...]
[junit4] Suite: org.apache.solr.cloud.TestPullReplicaErrorHandling
[junit4] 2> Creating dataDir: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/init-core-data-001
[junit4] 2> 492280 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 4 servers in /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001
[junit4] 2> 492280 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 492280 INFO (Thread-680) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 492280 INFO (Thread-680) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 492282 ERROR (Thread-680) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 492381 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:48034
[junit4] 2> 492384 INFO (zkConnectionManagerCallback-1493-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492390 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 492390 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@122a96fa{/solr,null,AVAILABLE}
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@16df55b2{HTTP/1.1,[http/1.1]}{127.0.0.1:54072}
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.e.j.s.Server Started @492655ms
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=54072}
[junit4] 2> 492391 ERROR (jetty-launcher-1490-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 492391 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:32.853Z
[junit4] 2> 492398 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 492398 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 492398 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 492400 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 492400 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 492400 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 492401 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@72ab2ad8{/solr,null,AVAILABLE}
[junit4] 2> 492401 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 492401 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 492401 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 492401 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.AbstractConnector Started ServerConnector@1170c69{HTTP/1.1,[http/1.1]}{127.0.0.1:41764}
[junit4] 2> 492401 INFO (jetty-launcher-1490-thread-3) [ ] o.e.j.s.Server Started @492666ms
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=41764}
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d6de04f{/solr,null,AVAILABLE}
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@169d9be7{/solr,null,AVAILABLE}
[junit4] 2> 492402 ERROR (jetty-launcher-1490-thread-3) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.AbstractConnector Started ServerConnector@6a839dcc{HTTP/1.1,[http/1.1]}{127.0.0.1:65207}
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-4) [ ] o.e.j.s.Server Started @492666ms
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:32.864Z
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=65207}
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@4d1c6381{HTTP/1.1,[http/1.1]}{127.0.0.1:62293}
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-2) [ ] o.e.j.s.Server Started @492666ms
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=62293}
[junit4] 2> 492402 ERROR (jetty-launcher-1490-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 492402 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:32.865Z
[junit4] 2> 492402 ERROR (jetty-launcher-1490-thread-4) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 492403 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:32.865Z
[junit4] 2> 492403 INFO (zkConnectionManagerCallback-1495-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492408 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 492409 INFO (zkConnectionManagerCallback-1499-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492409 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 492411 INFO (zkConnectionManagerCallback-1497-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492414 INFO (zkConnectionManagerCallback-1501-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492415 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 492415 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 492415 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f661d8af0001, likely client has closed socket
[junit4] 2> 492415 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f661d8af0003, likely client has closed socket
[junit4] 2> 492588 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f661d8af0004, likely client has closed socket
[junit4] 2> 492694 INFO (jetty-launcher-1490-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 492731 INFO (jetty-launcher-1490-thread-3) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 492733 INFO (zkConnectionManagerCallback-1505-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492736 INFO (zkConnectionManagerCallback-1511-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492772 INFO (zkConnectionManagerCallback-1509-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492822 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:54072_solr
[junit4] 2> 492823 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.Overseer Overseer (id=72328494145536007-127.0.0.1:54072_solr-n_0000000000) starting
[junit4] 2> 492834 INFO (zkConnectionManagerCallback-1520-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492844 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:41764_solr
[junit4] 2> 492863 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 492863 INFO (zkCallback-1510-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 492868 INFO (zkConnectionManagerCallback-1527-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492869 INFO (zkConnectionManagerCallback-1518-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 492870 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 492871 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 492880 INFO (OverseerStateUpdate-72328494145536007-127.0.0.1:54072_solr-n_0000000000) [n:127.0.0.1:54072_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:54072_solr
[junit4] 2> 492882 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 492882 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54072_solr
[junit4] 2> 492895 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 492918 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54072.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 492924 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54072.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 492924 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_54072.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 492925 INFO (jetty-launcher-1490-thread-1) [n:127.0.0.1:54072_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node1/.
[junit4] 2> 492926 INFO (zkCallback-1517-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 492927 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 492944 INFO (zkCallback-1510-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 492955 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 492956 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 492956 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 492971 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41764.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 492980 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41764.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 492980 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_41764.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 492980 INFO (jetty-launcher-1490-thread-3) [n:127.0.0.1:41764_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node3/.
[junit4] 2> 493099 INFO (jetty-launcher-1490-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 493107 INFO (zkConnectionManagerCallback-1533-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 493180 INFO (zkConnectionManagerCallback-1535-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 493191 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 493195 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 493195 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:62293_solr
[junit4] 2> 493219 INFO (zkCallback-1534-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 493220 INFO (zkCallback-1510-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 493221 INFO (zkCallback-1526-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 493221 INFO (zkCallback-1517-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 493222 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
[junit4] 2> 493222 INFO (zkConnectionManagerCallback-1542-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 493224 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 493225 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 493225 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 493248 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62293.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 493254 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62293.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 493254 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_62293.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 493255 INFO (jetty-launcher-1490-thread-2) [n:127.0.0.1:62293_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node2/.
[junit4] 2> 493960 INFO (jetty-launcher-1490-thread-4) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 493961 INFO (zkConnectionManagerCallback-1547-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 493964 INFO (zkConnectionManagerCallback-1549-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 493975 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 493981 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 493981 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:65207_solr
[junit4] 2> 493987 INFO (zkCallback-1517-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 493987 INFO (zkCallback-1510-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 493987 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 493988 INFO (zkCallback-1526-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 493988 INFO (zkCallback-1541-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 493988 INFO (zkCallback-1534-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 493989 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494009 INFO (zkConnectionManagerCallback-1556-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494011 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 494011 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 494012 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 494031 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65207.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 494036 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65207.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 494036 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_65207.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 494037 INFO (jetty-launcher-1490-thread-4) [n:127.0.0.1:65207_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node4/.
[junit4] 2> 494064 INFO (zkConnectionManagerCallback-1559-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494068 INFO (zkConnectionManagerCallback-1564-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494070 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 494071 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 494082 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@16df55b2{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 494083 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=1119756587
[junit4] 2> 494083 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 494083 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2140aa36: rootName = solr_54072, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@667d58f2
[junit4] 2> 494089 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 494089 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@7e6bd869: rootName = solr_54072, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@f5d8cd8
[junit4] 2> 494093 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 494093 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@c427435: rootName = solr_54072, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@544f2e84
[junit4] 2> 494095 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:54072_solr
[junit4] 2> 494096 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 494098 INFO (zkCallback-1510-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494098 INFO (zkCallback-1526-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494098 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494098 INFO (zkCallback-1517-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494099 INFO (zkCallback-1555-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494099 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494099 INFO (zkCallback-1534-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494099 INFO (zkCallback-1563-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494101 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.Overseer Overseer (id=72328494145536007-127.0.0.1:54072_solr-n_0000000000) closing
[junit4] 2> 494102 INFO (zkCallback-1541-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494102 INFO (OverseerStateUpdate-72328494145536007-127.0.0.1:54072_solr-n_0000000000) [n:127.0.0.1:54072_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:54072_solr
[junit4] 2> 494103 WARN (OverseerAutoScalingTriggerThread-72328494145536007-127.0.0.1:54072_solr-n_0000000000) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 494106 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f661d8af0007, likely client has closed socket
[junit4] 2> 494108 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@122a96fa{/solr,null,UNAVAILABLE}
[junit4] 2> 494108 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 494109 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:41764_solr
[junit4] 2> 494111 INFO (zkCallback-1519-thread-1) [n:127.0.0.1:41764_solr ] o.a.s.c.Overseer Overseer (id=72328494145536008-127.0.0.1:41764_solr-n_0000000001) starting
[junit4] 2> 494112 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 494112 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 494112 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 494112 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 494112 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@21da0bf5{/solr,null,AVAILABLE}
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Started ServerConnector@21cde263{HTTP/1.1,[http/1.1]}{127.0.0.1:36050}
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.Server Started @494379ms
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=38517}
[junit4] 2> 494114 ERROR (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 494114 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:34.576Z
[junit4] 2> 494121 INFO (OverseerStateUpdate-72328494145536008-127.0.0.1:41764_solr-n_0000000001) [n:127.0.0.1:41764_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:41764_solr
[junit4] 2> 494123 INFO (zkConnectionManagerCallback-1566-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494124 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 494708 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 494710 INFO (zkConnectionManagerCallback-1570-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494713 INFO (zkConnectionManagerCallback-1572-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494724 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 494731 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 494731 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38517_solr
[junit4] 2> 494733 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494734 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494735 INFO (zkCallback-1526-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494735 INFO (zkCallback-1541-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494735 INFO (zkCallback-1534-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494735 INFO (zkCallback-1555-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494736 INFO (zkCallback-1571-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494737 INFO (zkCallback-1563-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 494756 INFO (zkConnectionManagerCallback-1579-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494761 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 494763 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 494763 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 494777 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38517.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 494783 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38517.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 494783 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_38517.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 494784 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [n:127.0.0.1:38517_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node1/.
[junit4] 2> 494814 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.TestPullReplicaErrorHandling Adding proxy for URL: http://127.0.0.1:36050/solr. Proxy: http://127.0.0.1:38517/solr
[junit4] 2> 494829 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@1170c69{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 494829 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=311845773
[junit4] 2> 494830 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 494830 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@4f29691: rootName = solr_41764, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@1f3370f9
[junit4] 2> 494835 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 494835 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@40aad2b3: rootName = solr_41764, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@f5d8cd8
[junit4] 2> 494838 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 494838 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5fe837be: rootName = solr_41764, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@544f2e84
[junit4] 2> 494840 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:41764_solr
[junit4] 2> 494841 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 494841 INFO (zkCallback-1519-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494842 INFO (zkCallback-1534-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494842 INFO (zkCallback-1571-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494842 INFO (zkCallback-1555-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494843 INFO (zkCallback-1563-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494843 INFO (zkCallback-1578-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494844 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494844 INFO (zkCallback-1526-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494844 INFO (zkCallback-1541-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 494845 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 494845 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.Overseer Overseer (id=72328494145536008-127.0.0.1:41764_solr-n_0000000001) closing
[junit4] 2> 494845 WARN (OverseerAutoScalingTriggerThread-72328494145536008-127.0.0.1:41764_solr-n_0000000001) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 494846 INFO (OverseerStateUpdate-72328494145536008-127.0.0.1:41764_solr-n_0000000001) [n:127.0.0.1:41764_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:41764_solr
[junit4] 2> 494854 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@72ab2ad8{/solr,null,UNAVAILABLE}
[junit4] 2> 494855 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 494863 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 494863 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 494863 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 494864 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 494864 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@75b48f34{/solr,null,AVAILABLE}
[junit4] 2> 494864 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Started ServerConnector@d4c9657{HTTP/1.1,[http/1.1]}{127.0.0.1:61375}
[junit4] 2> 494864 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.Server Started @495129ms
[junit4] 2> 494865 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=51108}
[junit4] 2> 494865 ERROR (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 494866 INFO (zkCallback-1534-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:62293_solr
[junit4] 2> 494867 INFO (zkCallback-1534-thread-1) [n:127.0.0.1:62293_solr ] o.a.s.c.Overseer Overseer (id=72328494145536012-127.0.0.1:62293_solr-n_0000000002) starting
[junit4] 2> 494873 INFO (OverseerStateUpdate-72328494145536012-127.0.0.1:62293_solr-n_0000000002) [n:127.0.0.1:62293_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:62293_solr
[junit4] 2> 494876 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 494876 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 494876 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 494876 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 494876 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:35.338Z
[junit4] 2> 494882 INFO (zkConnectionManagerCallback-1582-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 494883 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 495838 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 495840 INFO (zkConnectionManagerCallback-1586-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 495844 INFO (zkConnectionManagerCallback-1588-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 495851 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 495856 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 495856 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:51108_solr
[junit4] 2> 495863 INFO (zkCallback-1541-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495863 INFO (zkCallback-1587-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495863 INFO (zkCallback-1571-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495864 INFO (zkCallback-1534-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495865 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495865 INFO (zkCallback-1563-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495866 INFO (zkCallback-1555-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495866 INFO (zkCallback-1578-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 495901 INFO (zkConnectionManagerCallback-1595-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 495903 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 495904 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 495905 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 495919 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51108.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 495932 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51108.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 495932 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_51108.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 495932 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node3/.
[junit4] 2> 495987 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.TestPullReplicaErrorHandling Adding proxy for URL: http://127.0.0.1:61375/solr. Proxy: http://127.0.0.1:51108/solr
[junit4] 2> 495988 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@4d1c6381{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 495988 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=182281509
[junit4] 2> 495989 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 495989 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@6e9fff50: rootName = solr_62293, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@11c27c02
[junit4] 2> 495994 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 495994 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@27586e22: rootName = solr_62293, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@f5d8cd8
[junit4] 2> 495996 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 495996 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1145709b: rootName = solr_62293, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@544f2e84
[junit4] 2> 495998 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:62293_solr
[junit4] 2> 495999 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.cluster, tag=null
[junit4] 2> 495999 INFO (zkCallback-1534-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496000 INFO (zkCallback-1541-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496000 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496000 INFO (zkCallback-1555-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496000 INFO (zkCallback-1571-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496000 INFO (zkCallback-1594-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496001 INFO (zkCallback-1563-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496001 INFO (zkCallback-1578-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496002 INFO (zkCallback-1587-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (4) -> (3)
[junit4] 2> 496003 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.Overseer Overseer (id=72328494145536012-127.0.0.1:62293_solr-n_0000000002) closing
[junit4] 2> 496003 WARN (OverseerAutoScalingTriggerThread-72328494145536012-127.0.0.1:62293_solr-n_0000000002) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 496004 INFO (OverseerStateUpdate-72328494145536012-127.0.0.1:62293_solr-n_0000000002) [n:127.0.0.1:62293_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:62293_solr
[junit4] 2> 496007 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f661d8af000c, likely client has closed socket
[junit4] 2> 496010 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@169d9be7{/solr,null,UNAVAILABLE}
[junit4] 2> 496010 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 496010 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:65207_solr
[junit4] 2> 496012 INFO (zkCallback-1548-thread-1) [n:127.0.0.1:65207_solr ] o.a.s.c.Overseer Overseer (id=72328494145536015-127.0.0.1:65207_solr-n_0000000003) starting
[junit4] 2> 496015 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.session node0 Scavenging every 660000ms
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@243425ce{/solr,null,AVAILABLE}
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Started ServerConnector@34a3cddb{HTTP/1.1,[http/1.1]}{127.0.0.1:64870}
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.Server Started @496281ms
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=39684}
[junit4] 2> 496016 ERROR (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 496016 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-13T09:25:36.478Z
[junit4] 2> 496021 INFO (zkConnectionManagerCallback-1598-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 496023 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 496023 INFO (OverseerStateUpdate-72328494145536015-127.0.0.1:65207_solr-n_0000000003) [n:127.0.0.1:65207_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:65207_solr
[junit4] 2> 496484 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:48034/solr
[junit4] 2> 496484 INFO (zkConnectionManagerCallback-1602-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 496491 INFO (zkConnectionManagerCallback-1604-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 496503 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
[junit4] 2> 496510 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 496510 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39684_solr
[junit4] 2> 496513 INFO (zkCallback-1571-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496513 INFO (zkCallback-1587-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496513 INFO (zkCallback-1578-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496513 INFO (zkCallback-1555-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496513 INFO (zkCallback-1563-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496513 INFO (zkCallback-1603-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496514 INFO (zkCallback-1548-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496515 INFO (zkCallback-1594-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
[junit4] 2> 496530 INFO (zkConnectionManagerCallback-1611-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 496532 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (4)
[junit4] 2> 496537 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:48034/solr ready
[junit4] 2> 496538 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 496551 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39684.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 496557 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39684.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 496557 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_39684.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@474a80f9
[junit4] 2> 496558 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J0/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-001/tempDir-001/node2/.
[junit4] 2> 496586 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.TestPullReplicaErrorHandling Adding proxy for URL: http://127.0.0.1:64870/solr. Proxy: http://127.0.0.1:39684/solr
[junit4] 2> 496587 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.e.j.s.AbstractConnector Stopped ServerConnector@6a839dcc{HTTP/1.1,[http/1.1]}{127.0.0.1:0}
[junit4] 2> 496587 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=2128865983
[junit4] 2> 496587 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 496588 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@395bfe8f: rootName = solr_65207, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@17b8fb47
[junit4] 2> 496592 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 496592 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@513fb7df: rootName = solr_65207, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@f5d8cd8
[junit4] 2> 496594 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 496594 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@5da2bb0b: rootName = solr_65207, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@544f2e84
[junit4] 2> 496597 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkController Remove node as live in ZooKeeper:/live_nodes/127.0.0.1:65207_solr
[junit4] 2> 496598 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.m.SolrMetricManager Closing
[...truncated too long message...]
FO (jetty-closer-286-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72328935818723349-127.0.0.1:39484_solr-n_0000000004) closing
[junit4] 2> 1898739 INFO (OverseerStateUpdate-72328935818723349-127.0.0.1:39484_solr-n_0000000004) [n:127.0.0.1:39484_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:39484_solr
[junit4] 2> 1898741 INFO (zkCallback-366-thread-6) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (0)
[junit4] 2> 1898741 INFO (zkCallback-382-thread-3) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (0)
[junit4] 2> 1898742 INFO (zkCallback-414-thread-3) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (0)
[junit4] 2> 1898742 WARN (OverseerAutoScalingTriggerThread-72328935818723349-127.0.0.1:39484_solr-n_0000000004) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 1898743 INFO (zkCallback-373-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (0)
[junit4] 2> 1898743 INFO (zkCallback-421-thread-2) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (0)
[junit4] 2> 1898743 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f6c8ae750022, likely client has closed socket
[junit4] 2> 1898743 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f6c8ae75001a, likely client has closed socket
[junit4] 2> 1898744 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f6c8ae750016, likely client has closed socket
[junit4] 2> 1898748 INFO (jetty-closer-286-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1a3c23cb{/solr,null,UNAVAILABLE}
[junit4] 2> 1898749 INFO (jetty-closer-286-thread-2) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 1898750 INFO (jetty-closer-286-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@19c6d781{/solr,null,UNAVAILABLE}
[junit4] 2> 1898751 INFO (jetty-closer-286-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 1898753 INFO (zkCallback-414-thread-3) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:39876_solr
[junit4] 2> 1898753 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxn Unable to read additional data from client sessionid 0x100f6c8ae750021, likely client has closed socket
[junit4] 2> 1898754 INFO (jetty-closer-286-thread-4) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@1ef72059{/solr,null,UNAVAILABLE}
[junit4] 2> 1898755 INFO (jetty-closer-286-thread-4) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 1898756 ERROR (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1898757 INFO (SUITE-TestPullReplicaErrorHandling-seed#[BAD8350446A6B372]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42151 42151
[junit4] 2> 1899560 WARN (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [ ] o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
[junit4] 2> java.nio.channels.CancelledKeyException: null
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) ~[?:1.8.0_172]
[junit4] 2> at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87) ~[?:1.8.0_172]
[junit4] 2> at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:205) [zookeeper-3.4.11.jar:3.4.11-37e277162d567b55a07d1755f0b31c32e93c01a0]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_172]
[junit4] 2> 1899561 INFO (Thread-176) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:42151 42151
[junit4] 2> 1899563 WARN (Thread-176) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 18 /solr/aliases.json
[junit4] 2> 17 /solr/clusterprops.json
[junit4] 2> 9 /solr/security.json
[junit4] 2> 9 /solr/configs/conf
[junit4] 2> 7 /solr/collections/pull_replica_error_handling_test_cant_connect_to_leader/terms/shard1
[junit4] 2> 7 /solr/collections/pull_replica_error_handling_test_pull_replica_disconnects_from_zoo_keeper/terms/shard1
[junit4] 2> 7 /solr/collections/pull_replica_error_handling_test_cant_connect_to_pull_replica/terms/shard1
[junit4] 2> 7 /solr/collections/pull_replica_error_handling_test_cant_connect_to_pull_replica/terms/shard2
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 18 /solr/clusterstate.json
[junit4] 2> 10 /solr/clusterprops.json
[junit4] 2> 7 /solr/collections/pull_replica_error_handling_test_pull_replica_disconnects_from_zoo_keeper/state.json
[junit4] 2> 4 /solr/collections/pull_replica_error_handling_test_cant_connect_to_pull_replica/state.json
[junit4] 2> 3 /solr/overseer_elect/election/72328935818723338-127.0.0.1:41930_solr-n_0000000001
[junit4] 2> 2 /solr/collections/pull_replica_error_handling_test_cant_connect_to_leader/state.json
[junit4] 2> 2 /solr/overseer_elect/election/72328935818723353-127.0.0.1:46758_solr-n_0000000005
[junit4] 2> 2 /solr/autoscaling.json
[junit4] 2> 2 /solr/overseer_elect/election/72328935818723337-127.0.0.1:40967_solr-n_0000000000
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 18 /solr/collections
[junit4] 2> 17 /solr/live_nodes
[junit4] 2> 2 /solr/overseer/queue
[junit4] 2> 2 /solr/autoscaling/events/.scheduled_maintenance
[junit4] 2> 2 /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 2 /solr/overseer/collection-queue-work
[junit4] 2> 2 /solr/overseer/queue-work
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestPullReplicaErrorHandling_BAD8350446A6B372-003
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {fieldName_s=Lucene50(blocksize=128), id=Lucene50(blocksize=128)}, docValues:{}, maxPointsInLeafNode=314, maxMBSortInHeap=6.059464951906051, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4fe9e331), locale=he, timezone=America/Guayaquil
[junit4] 2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=3,threads=1,free=125397992,total=269484032
[junit4] 2> NOTE: All tests run in this JVM: [TestPullReplicaErrorHandling, TestPullReplicaErrorHandling, TestPullReplicaErrorHandling]
[junit4] Completed [5/5 (5!)] on J1 in 636.35s, 3 tests, 1 error <<< FAILURES!
[junit4]
[junit4]
[junit4] Tests with failures [seed: BAD8350446A6B372]:
[junit4] - org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
[junit4] - org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
[junit4] - org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
[junit4] - org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
[junit4] - org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader
[junit4]
[junit4]
[junit4] JVM J0: 0.77 .. 1271.97 = 1271.20s
[junit4] JVM J1: 0.72 .. 1900.78 = 1900.07s
[junit4] Execution time total: 31 minutes 40 seconds
[junit4] Tests summary: 5 suites, 15 tests, 5 errors
BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1567: The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1092: There were test failures: 5 suites, 15 tests, 5 errors [seed: BAD8350446A6B372]
Total time: 31 minutes 42 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 5/5 failed: org.apache.solr.cloud.TestPullReplicaErrorHandling
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=4a5b914eaa8683009191748bf6c0b1be14d59661, workspace=/export/home/jenkins/workspace/Lucene-Solr-master-Solaris
[WARNINGS] Computing warning deltas based on reference build #2060
Recording test results
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/export/home/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2