You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2018/09/21 04:17:27 UTC
[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-11-ea+28) - Build
# 22895 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22895/
Java: 64bit/jdk-11-ea+28 -XX:-UseCompressedOops -XX:+UseG1GC
31 tests failed.
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Could not find collection : AutoscalingHistoryHandlerTest_collection
Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : AutoscalingHistoryHandlerTest_collection
at __randomizedtesting.SeedInfo.seed([823E3160365E8A1E]:0)
at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:258)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.waitForRecovery(AutoscalingHistoryHandlerTest.java:403)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)
FAILED: junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
Error Message:
Could not find collection : AutoscalingHistoryHandlerTest_collection
Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : AutoscalingHistoryHandlerTest_collection
at __randomizedtesting.SeedInfo.seed([823E3160365E8A1E]:0)
at org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
at org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:258)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.waitForRecovery(AutoscalingHistoryHandlerTest.java:403)
at org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
20 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest: 1) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 2) Thread[id=1578, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 3) Thread[id=1571, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 4) Thread[id=2601, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 5) Thread[id=2595, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 6) Thread[id=385, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 7) Thread[id=1572, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 8) Thread[id=2598, name=zkConnectionManagerCallback-1268-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 9) Thread[id=392, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 10) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 11) Thread[id=1577, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 12) Thread[id=2596, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 13) Thread[id=2597, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 14) Thread[id=2602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 15) Thread[id=387, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 16) Thread[id=391, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 17) Thread[id=1584, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 18) Thread[id=1585, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 19) Thread[id=1573, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 20) Thread[id=1574, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest:
1) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
2) Thread[id=1578, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
3) Thread[id=1571, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
4) Thread[id=2601, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=2595, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
6) Thread[id=385, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
7) Thread[id=1572, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
8) Thread[id=2598, name=zkConnectionManagerCallback-1268-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
9) Thread[id=392, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
10) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
11) Thread[id=1577, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
12) Thread[id=2596, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
13) Thread[id=2597, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
14) Thread[id=2602, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
15) Thread[id=387, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
16) Thread[id=391, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
17) Thread[id=1584, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
18) Thread[id=1585, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
19) Thread[id=1573, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
20) Thread[id=1574, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 2) Thread[id=2596, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 3) Thread[id=1572, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 4) Thread[id=2598, name=zkConnectionManagerCallback-1268-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 5) Thread[id=1574, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 6) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
2) Thread[id=2596, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
3) Thread[id=1572, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
4) Thread[id=2598, name=zkConnectionManagerCallback-1268-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=1574, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
6) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:39341), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
20 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest: 1) Thread[id=941, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 2) Thread[id=3153, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 3) Thread[id=2129, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 4) Thread[id=3157, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 5) Thread[id=946, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 6) Thread[id=940, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 7) Thread[id=947, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 8) Thread[id=943, name=zkConnectionManagerCallback-525-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 9) Thread[id=2146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 10) Thread[id=3154, name=zkConnectionManagerCallback-1617-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 11) Thread[id=942, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 12) Thread[id=2135, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 13) Thread[id=3151, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 14) Thread[id=3152, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 15) Thread[id=2143, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 16) Thread[id=2128, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 17) Thread[id=2134, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 18) Thread[id=2130, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 19) Thread[id=2131, name=zkConnectionManagerCallback-1110-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 20) Thread[id=3158, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 20 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest:
1) Thread[id=941, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
2) Thread[id=3153, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
3) Thread[id=2129, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
4) Thread[id=3157, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=946, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
6) Thread[id=940, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
7) Thread[id=947, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
8) Thread[id=943, name=zkConnectionManagerCallback-525-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
9) Thread[id=2146, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
10) Thread[id=3154, name=zkConnectionManagerCallback-1617-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
11) Thread[id=942, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
12) Thread[id=2135, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
13) Thread[id=3151, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
14) Thread[id=3152, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
15) Thread[id=2143, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
16) Thread[id=2128, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
17) Thread[id=2134, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
18) Thread[id=2130, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
19) Thread[id=2131, name=zkConnectionManagerCallback-1110-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
20) Thread[id=3158, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=941, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 2) Thread[id=2129, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054) 3) Thread[id=3152, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 4) Thread[id=943, name=zkConnectionManagerCallback-525-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 5) Thread[id=2131, name=zkConnectionManagerCallback-1110-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 6) Thread[id=3154, name=zkConnectionManagerCallback-1617-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=941, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
2) Thread[id=2129, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1054)
3) Thread[id=3152, name=TEST-StreamDecoratorTest.testParallelHavingStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:45005), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
4) Thread[id=943, name=zkConnectionManagerCallback-525-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=2131, name=zkConnectionManagerCallback-1110-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
6) Thread[id=3154, name=zkConnectionManagerCallback-1617-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
14 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest: 1) Thread[id=1571, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230) at app//org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1249) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1173) 2) Thread[id=1583, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 3) Thread[id=1577, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 4) Thread[id=1584, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 5) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 6) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 7) Thread[id=392, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 8) Thread[id=1573, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 9) Thread[id=385, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 10) Thread[id=391, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 11) Thread[id=387, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 12) Thread[id=1572, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502) 13) Thread[id=1576, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834) 14) Thread[id=1570, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 14 threads leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest:
1) Thread[id=1571, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:230)
at app//org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1249)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1173)
2) Thread[id=1583, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
3) Thread[id=1577, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
4) Thread[id=1584, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
5) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
6) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
7) Thread[id=392, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
8) Thread[id=1573, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
9) Thread[id=385, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
10) Thread[id=391, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
11) Thread[id=387, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
12) Thread[id=1572, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-EventThread, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at app//org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:502)
13) Thread[id=1576, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
14) Thread[id=1570, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
There are still zombie threads that couldn't be terminated: 1) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 2) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834) 3) Thread[id=1571, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105) at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000) at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063) 4) Thread[id=1573, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/jdk.internal.misc.Unsafe.park(Native Method) at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081) at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433) at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054) at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114) at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
1) Thread[id=386, name=TEST-StreamDecoratorTest.testParallelExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
2) Thread[id=388, name=zkConnectionManagerCallback-176-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
3) Thread[id=1571, name=TEST-StreamDecoratorTest.testExecutorStream-seed#[41DD9F98675E91C0]-SendThread(127.0.0.1:37847), state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:105)
at app//org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1000)
at app//org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1063)
4) Thread[id=1573, name=zkConnectionManagerCallback-761-thread-1, state=WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at java.base@11/java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:433)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1054)
at java.base@11/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1114)
at java.base@11/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base@11/java.lang.Thread.run(Thread.java:834)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest: 1) Thread[id=2108, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest] at java.base@11/java.lang.Thread.sleep(Native Method) at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66) at java.base@11/java.lang.Thread.run(Thread.java:834)
Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest:
1) Thread[id=2108, name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamDecoratorTest]
at java.base@11/java.lang.Thread.sleep(Native Method)
at app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@11/java.lang.Thread.run(Thread.java:834)
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0]:0)
FAILED: org.apache.solr.client.solrj.io.stream.StreamDecoratorTest.testParallelShuffleStream
Error Message:
java.util.concurrent.ExecutionException: java.io.IOException: --> https://127.0.0.1:33597/solr/collection1_collection_shard1_replica_n1/:java.util.concurrent.ExecutionException: java.io.IOException: params q=*:*&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&partitionKeys=a_f&qt=/export&distrib=false
Stack Trace:
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: --> https://127.0.0.1:33597/solr/collection1_collection_shard1_replica_n1/:java.util.concurrent.ExecutionException: java.io.IOException: params q=*:*&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&partitionKeys=a_f&qt=/export&distrib=false
at __randomizedtesting.SeedInfo.seed([41DD9F98675E91C0:C30CEB5A3252C8B8]:0)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:400)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream.open(CloudSolrStream.java:275)
at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest.getTuples(StreamDecoratorTest.java:3953)
at org.apache.solr.client.solrj.io.stream.StreamDecoratorTest.testParallelShuffleStream(StreamDecoratorTest.java:1368)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: java.io.IOException: --> https://127.0.0.1:33597/solr/collection1_collection_shard1_replica_n1/:java.util.concurrent.ExecutionException: java.io.IOException: params q=*:*&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&partitionKeys=a_f&qt=/export&distrib=false
at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream.openStreams(CloudSolrStream.java:394)
... 41 more
Caused by: java.io.IOException: --> https://127.0.0.1:33597/solr/collection1_collection_shard1_replica_n1/:java.util.concurrent.ExecutionException: java.io.IOException: params q=*:*&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&partitionKeys=a_f&qt=/export&distrib=false
at org.apache.solr.client.solrj.io.stream.SolrStream.read(SolrStream.java:215)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream$TupleWrapper.next(CloudSolrStream.java:484)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:507)
at org.apache.solr.client.solrj.io.stream.CloudSolrStream$StreamOpener.call(CloudSolrStream.java:494)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
... 1 more
FAILED: org.apache.solr.client.solrj.io.stream.StreamDecoratorTest.testParallelShuffleStream
Error Message:
java.util.concurrent.ExecutionException: java.io.IOException: --> https://127.0.0.1:35239/solr/collection1_collection_shard2_replica_n2/:java.util.concurrent.ExecutionException: java.io.IOException: params q=*:*&fl=id,a_s,a_i,a_f&sort=a_f+asc,+a_i+asc&partitionKeys=a_f&qt=/export&distrib=false
Stack Trace:
java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: -->
[...truncated too long message...]
http/1.1]}{127.0.0.1:0}
[junit4] 2> 69046 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.c.CoreContainer Shutting down CoreContainer instance=90917264
[junit4] 2> 69046 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.node, tag=null
[junit4] 2> 69046 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@35b51803: rootName = null, domain = solr.node, service url = null, agent id = null] for registry solr.node / com.codahale.metrics.MetricRegistry@42a3b280
[junit4] 2> 69054 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jvm, tag=null
[junit4] 2> 69054 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@401dfea2: rootName = null, domain = solr.jvm, service url = null, agent id = null] for registry solr.jvm / com.codahale.metrics.MetricRegistry@10669c6e
[junit4] 2> 69058 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.jetty, tag=null
[junit4] 2> 69058 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@20b7c91e: rootName = null, domain = solr.jetty, service url = null, agent id = null] for registry solr.jetty / com.codahale.metrics.MetricRegistry@1f5764b4
[junit4] 2> 69059 INFO (coreCloseExecutor-386-thread-1) [ x:collection1] o.a.s.c.SolrCore [collection1] CLOSING SolrCore org.apache.solr.core.SolrCore@17be9610
[junit4] 2> 69059 INFO (coreCloseExecutor-386-thread-1) [ x:collection1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1, tag=17be9610
[junit4] 2> 69059 INFO (coreCloseExecutor-386-thread-1) [ x:collection1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@1b17ead9: rootName = null, domain = solr.core.collection1, service url = null, agent id = null] for registry solr.core.collection1 / com.codahale.metrics.MetricRegistry@73a0e0fa
[junit4] 2> 69069 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@165d55b{/,null,UNAVAILABLE}
[junit4] 2> 69069 INFO (TEST-DistributedMLTComponentTest.test-seed#[823E3160365E8A1E]) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=DistributedMLTComponentTest -Dtests.method=test -Dtests.seed=823E3160365E8A1E -Dtests.multiplier=3 -Dtests.slow=true -Dtests.badapples=true -Dtests.locale=sl -Dtests.timezone=Asia/Kashgar -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
[junit4] ERROR 2.70s J2 | DistributedMLTComponentTest.test <<<
[junit4] > Throwable #1: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at https://127.0.0.1:39415//collection1: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: https://127.0.0.1:39415//collection1
[junit4] > at __randomizedtesting.SeedInfo.seed([823E3160365E8A1E:A6A0EBA98A2E7E6]:0)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
[junit4] > at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
[junit4] > at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
[junit4] > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:942)
[junit4] > at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:957)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.queryServer(BaseDistributedSearchTestCase.java:559)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:611)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:589)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase.query(BaseDistributedSearchTestCase.java:568)
[junit4] > at org.apache.solr.handler.component.DistributedMLTComponentTest.test(DistributedMLTComponentTest.java:148)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4] > at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[junit4] > at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4] > at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
[junit4] > at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:983)
[junit4] > at java.base/java.lang.Thread.run(Thread.java:834)
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J2/temp/solr.handler.component.DistributedMLTComponentTest_823E3160365E8A1E-003
[junit4] 2> NOTE: test params are: codec=FastCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=FAST, chunkSize=3, maxDocsPerChunk=6, blockSize=733), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=FAST, chunkSize=3, blockSize=733)), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5e5a9d04), locale=sl, timezone=Asia/Kashgar
[junit4] 2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 11 (64-bit)/cpus=8,threads=1,free=196007288,total=536870912
[junit4] 2> NOTE: All tests run in this JVM: [DistributedMLTComponentTest, DistributedMLTComponentTest, AutoscalingHistoryHandlerTest, DistributedMLTComponentTest, DistributedMLTComponentTest, DistributedMLTComponentTest]
[junit4] Completed [8/10 (4!)] on J2 in 2.72s, 1 test, 1 error <<< FAILURES!
[junit4]
[junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[junit4] OK 32.4s J1 | AutoscalingHistoryHandlerTest.testHistory
[junit4] Completed [9/10 (4!)] on J1 in 40.51s, 1 test
[junit4]
[junit4] Suite: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[junit4] OK 32.6s J0 | AutoscalingHistoryHandlerTest.testHistory
[junit4] Completed [10/10 (4!)] on J0 in 40.69s, 1 test
[junit4]
[junit4]
[junit4] Tests with failures [seed: 823E3160365E8A1E]:
[junit4] - org.apache.solr.handler.component.DistributedMLTComponentTest.test
[junit4] - org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest (suite)
[junit4] - org.apache.solr.handler.component.DistributedMLTComponentTest.test
[junit4] - org.apache.solr.handler.component.DistributedMLTComponentTest.test
[junit4]
[junit4]
[junit4] JVM J0: 0.69 .. 98.84 = 98.15s
[junit4] JVM J1: 0.69 .. 90.86 = 90.17s
[junit4] JVM J2: 0.69 .. 69.56 = 68.87s
[junit4] Execution time total: 1 minute 38 seconds
[junit4] Tests summary: 10 suites, 9 tests, 1 suite-level error, 3 errors
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1092: There were test failures: 10 suites, 9 tests, 1 suite-level error, 3 errors [seed: 823E3160365E8A1E]
Total time: 1 minute 40 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 1/5 failed: org.apache.solr.client.solrj.io.stream.StreamingTest
[repro] 1/5 failed: org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[repro] 3/5 failed: org.apache.solr.client.solrj.io.stream.StreamDecoratorTest
[repro] 3/5 failed: org.apache.solr.handler.component.DistributedMLTComponentTest
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=d7e97fb7f84d8613683a080610f177b7cae2b31a, workspace=/var/lib/jenkins/workspace/Lucene-Solr-master-Linux
[WARNINGS] Computing warning deltas based on reference build #22894
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_172) - Build #
22896 - Still Unstable!
Posted by Policeman Jenkins Server <je...@thetaphi.de>.
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/22896/
Java: 64bit/jdk1.8.0_172 -XX:-UseCompressedOops -XX:+UseSerialGC
2 tests failed.
FAILED: org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
Error Message:
Stack Trace:
java.lang.AssertionError
at __randomizedtesting.SeedInfo.seed([75CA6A86654D6364:F6BC3574B3346DC5]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest(TestCloudRecovery.java:201)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
FAILED: org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest
Error Message:
expected:<4> but was:<0>
Stack Trace:
java.lang.AssertionError: expected:<4> but was:<0>
at __randomizedtesting.SeedInfo.seed([75CA6A86654D6364:13A8BC3216924EB]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at org.apache.solr.cloud.TestCloudRecovery.leaderRecoverFromLogOnStartupTest(TestCloudRecovery.java:113)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Build Log:
[...truncated 13976 lines...]
[junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
[junit4] 2> 1553580 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
[junit4] 2> Creating dataDir: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/init-core-data-001
[junit4] 2> 1553580 WARN (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=13 numCloses=13
[junit4] 2> 1553581 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
[junit4] 2> 1553582 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
[junit4] 2> 1553583 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001
[junit4] 2> 1553583 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
[junit4] 2> 1553583 INFO (Thread-5649) [ ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
[junit4] 2> 1553583 INFO (Thread-5649) [ ] o.a.s.c.ZkTestServer Starting server
[junit4] 2> 1553584 ERROR (Thread-5649) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 1553683 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.ZkTestServer start zk server on port:39083
[junit4] 2> 1553685 INFO (zkConnectionManagerCallback-4847-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1553687 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1553687 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.Server jetty-9.4.11.v20180605; built: 2018-06-05T18:24:03.829Z; git: d5fc0523cfa96bfebfbda19606cad384d772f04c; jvm 1.8.0_172-b11
[junit4] 2> 1553689 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1553689 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.session DefaultSessionIdManager workerName=node0
[junit4] 2> 1553689 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.session No SessionScavenger set, using defaults
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.session node0 Scavenging every 600000ms
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@3d0265dc{/solr,null,AVAILABLE}
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@122340b6{/solr,null,AVAILABLE}
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.AbstractConnector Started ServerConnector@52e421d8{HTTP/1.1,[http/1.1]}{127.0.0.1:43489}
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.AbstractConnector Started ServerConnector@222f931e{HTTP/1.1,[http/1.1]}{127.0.0.1:36761}
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.e.j.s.Server Started @1553721ms
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.e.j.s.Server Started @1553721ms
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36761}
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=43489}
[junit4] 2> 1553690 ERROR (jetty-launcher-4844-thread-1) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1553690 ERROR (jetty-launcher-4844-thread-2) [ ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.s.SolrDispatchFilter ___ _ Welcome to Apache Solr™ version 8.0.0
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port null
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: null
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-21T07:22:57.011Z
[junit4] 2> 1553690 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2018-09-21T07:22:57.011Z
[junit4] 2> 1553691 INFO (zkConnectionManagerCallback-4851-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1553691 INFO (zkConnectionManagerCallback-4850-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1553692 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1553692 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
[junit4] 2> 1553797 INFO (jetty-launcher-4844-thread-1) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39083/solr
[junit4] 2> 1553798 INFO (zkConnectionManagerCallback-4855-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1553799 INFO (zkConnectionManagerCallback-4857-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1553831 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36761_solr
[junit4] 2> 1553831 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.c.Overseer Overseer (id=72261427679002628-127.0.0.1:36761_solr-n_0000000000) starting
[junit4] 2> 1553834 INFO (zkConnectionManagerCallback-4864-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1553835 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39083/solr ready
[junit4] 2> 1553836 INFO (OverseerStateUpdate-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:36761_solr
[junit4] 2> 1553836 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36761_solr
[junit4] 2> 1553836 INFO (OverseerStateUpdate-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1553838 INFO (zkCallback-4863-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1553852 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1553875 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36761.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1553882 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36761.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1553882 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36761.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1553883 INFO (jetty-launcher-4844-thread-1) [n:127.0.0.1:36761_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node1/.
[junit4] 2> 1554007 INFO (jetty-launcher-4844-thread-2) [ ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39083/solr
[junit4] 2> 1554008 INFO (zkConnectionManagerCallback-4869-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1554009 INFO (zkConnectionManagerCallback-4871-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1554011 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
[junit4] 2> 1554012 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1554012 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43489_solr
[junit4] 2> 1554013 INFO (zkCallback-4856-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1554013 INFO (zkCallback-4863-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1554013 INFO (zkCallback-4870-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
[junit4] 2> 1554023 INFO (zkConnectionManagerCallback-4878-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1554024 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1554024 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39083/solr ready
[junit4] 2> 1554024 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
[junit4] 2> 1554036 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43489.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1554041 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43489.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1554041 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43489.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1554041 INFO (jetty-launcher-4844-thread-2) [n:127.0.0.1:43489_solr ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node2/.
[junit4] 2> 1554056 INFO (zkConnectionManagerCallback-4881-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1554058 INFO (zkConnectionManagerCallback-4886-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected
[junit4] 2> 1554059 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
[junit4] 2> 1554060 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39083/solr ready
[junit4] 2> 1554061 INFO (qtp1575998988-19701) [n:127.0.0.1:36761_solr ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params pullReplicas=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2 and sendToOCPQueue=true
[junit4] 2> 1554062 INFO (OverseerThreadFactory-6808-thread-1-processing-n:127.0.0.1:36761_solr) [n:127.0.0.1:36761_solr ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
[junit4] 2> 1554166 INFO (OverseerStateUpdate-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t1",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:43489/solr",
[junit4] 2> "type":"TLOG",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1554167 INFO (OverseerStateUpdate-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard1",
[junit4] 2> "core":"collection1_shard1_replica_t2",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:36761/solr",
[junit4] 2> "type":"TLOG",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1554168 INFO (OverseerStateUpdate-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t4",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:43489/solr",
[junit4] 2> "type":"TLOG",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1554169 INFO (OverseerStateUpdate-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.o.SliceMutator createReplica() {
[junit4] 2> "operation":"ADDREPLICA",
[junit4] 2> "collection":"collection1",
[junit4] 2> "shard":"shard2",
[junit4] 2> "core":"collection1_shard2_replica_t6",
[junit4] 2> "state":"down",
[junit4] 2> "base_url":"http://127.0.0.1:36761/solr",
[junit4] 2> "type":"TLOG",
[junit4] 2> "waitForFinalState":"false"}
[junit4] 2> 1554372 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr x:collection1_shard2_replica_t4] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1554372 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr x:collection1_shard1_replica_t2] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1554372 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr x:collection1_shard1_replica_t1] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1554372 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr x:collection1_shard1_replica_t2] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
[junit4] 2> 1554373 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr x:collection1_shard2_replica_t6] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG
[junit4] 2> 1555401 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1555402 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1555404 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1555406 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrConfig Using Lucene MatchVersion: 8.0.0
[junit4] 2> 1555411 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema [collection1_shard2_replica_t6] Schema name=minimal
[junit4] 2> 1555411 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema [collection1_shard2_replica_t4] Schema name=minimal
[junit4] 2> 1555428 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1555428 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t4' using configuration from collection collection1, trusted=true
[junit4] 2> 1555428 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43489.solr.core.collection1.shard2.replica_t4' (registry 'solr.core.collection1.shard2.replica_t4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1555428 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [[collection1_shard2_replica_t4] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node2/collection1_shard2_replica_t4], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node2/./collection1_shard2_replica_t4/data/]
[junit4] 2> 1555431 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1555432 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard2_replica_t6' using configuration from collection collection1, trusted=true
[junit4] 2> 1555432 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36761.solr.core.collection1.shard2.replica_t6' (registry 'solr.core.collection1.shard2.replica_t6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1555432 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema [collection1_shard1_replica_t1] Schema name=minimal
[junit4] 2> 1555432 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [[collection1_shard2_replica_t6] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node1/collection1_shard2_replica_t6], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node1/./collection1_shard2_replica_t6/data/]
[junit4] 2> 1555432 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema [collection1_shard1_replica_t2] Schema name=minimal
[junit4] 2> 1555434 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1555434 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t1' using configuration from collection collection1, trusted=true
[junit4] 2> 1555434 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
[junit4] 2> 1555435 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_43489.solr.core.collection1.shard1.replica_t1' (registry 'solr.core.collection1.shard1.replica_t1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1555435 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [[collection1_shard1_replica_t1] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node2/collection1_shard1_replica_t1], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node2/./collection1_shard1_replica_t1/data/]
[junit4] 2> 1555435 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.CoreContainer Creating SolrCore 'collection1_shard1_replica_t2' using configuration from collection collection1, trusted=true
[junit4] 2> 1555436 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36761.solr.core.collection1.shard1.replica_t2' (registry 'solr.core.collection1.shard1.replica_t2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@1f062360
[junit4] 2> 1555436 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [[collection1_shard1_replica_t2] ] Opening new SolrCore at [/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node1/collection1_shard1_replica_t2], dataDir=[/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001/tempDir-001/node1/./collection1_shard1_replica_t2/data/]
[junit4] 2> 1555495 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1555495 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1555496 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1555496 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1555497 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.SolrIndexSearcher Opening [Searcher@24c59c55[collection1_shard2_replica_t4] main]
[junit4] 2> 1555498 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1555498 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1555499 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1555499 INFO (searcherExecutor-6820-thread-1-processing-n:127.0.0.1:43489_solr x:collection1_shard2_replica_t4 c:collection1 s:shard2 r:core_node7) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [collection1_shard2_replica_t4] Registered new searcher Searcher@24c59c55[collection1_shard2_replica_t4] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1555499 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1612200887000760320
[junit4] 2> 1555502 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node7=0}, version=0}
[junit4] 2> 1555504 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=9999ms
[junit4] 2> 1555505 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1555505 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1555505 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1555505 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1555506 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1555506 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1555506 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1555506 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1555507 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.SolrIndexSearcher Opening [Searcher@7c172ea9[collection1_shard1_replica_t2] main]
[junit4] 2> 1555507 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.SolrIndexSearcher Opening [Searcher@6902fd9f[collection1_shard1_replica_t1] main]
[junit4] 2> 1555507 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1555507 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1555508 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1555508 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1555508 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1555508 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1555509 INFO (searcherExecutor-6822-thread-1-processing-n:127.0.0.1:43489_solr x:collection1_shard1_replica_t1 c:collection1 s:shard1 r:core_node3) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@6902fd9f[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1555509 INFO (searcherExecutor-6823-thread-1-processing-n:127.0.0.1:36761_solr x:collection1_shard1_replica_t2 c:collection1 s:shard1 r:core_node5) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SolrCore [collection1_shard1_replica_t2] Registered new searcher Searcher@7c172ea9[collection1_shard1_replica_t2] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1555509 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1612200887011246080
[junit4] 2> 1555509 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1612200887011246080
[junit4] 2> 1555511 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node5=0}, version=0}
[junit4] 2> 1555511 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkShardTerms Failed to save terms, version is not a match, retrying
[junit4] 2> 1555511 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=0, core_node5=0}, version=1}
[junit4] 2> 1555512 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1555512 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1555512 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:36761/solr/collection1_shard1_replica_t2/
[junit4] 2> 1555512 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t2 url=http://127.0.0.1:36761/solr START replicas=[http://127.0.0.1:43489/solr/collection1_shard1_replica_t1/] nUpdates=100
[junit4] 2> 1555513 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.PeerSync PeerSync: core=collection1_shard1_replica_t2 url=http://127.0.0.1:36761/solr DONE. We have no versions. sync failed.
[junit4] 2> 1555513 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
[junit4] 2> 1555513 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir=null defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
[junit4] 2> 1555513 INFO (qtp1734158036-19692) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.S.Request [collection1_shard1_replica_t1] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1555514 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1555514 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1555514 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1555514 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Hard AutoCommit: disabled
[junit4] 2> 1555514 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.CommitTracker Soft AutoCommit: disabled
[junit4] 2> 1555514 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkController collection1_shard1_replica_t2 stopping background replication from leader
[junit4] 2> 1555515 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:36761/solr/collection1_shard1_replica_t2/ shard1
[junit4] 2> 1555516 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.SolrIndexSearcher Opening [Searcher@60efc21f[collection1_shard2_replica_t6] main]
[junit4] 2> 1555516 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/config
[junit4] 2> 1555516 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/config
[junit4] 2> 1555517 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1555517 INFO (searcherExecutor-6821-thread-1-processing-n:127.0.0.1:36761_solr x:collection1_shard2_replica_t6 c:collection1 s:shard2 r:core_node8) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@60efc21f[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1555517 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1612200887019634688
[junit4] 2> 1555520 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node7=0, core_node8=0}, version=1}
[junit4] 2> 1555667 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1555668 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node5&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t2&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1296
[junit4] 2> 1555768 INFO (zkCallback-4856-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1556005 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
[junit4] 2> 1556005 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
[junit4] 2> 1556005 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/
[junit4] 2> 1556006 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:43489/solr START replicas=[http://127.0.0.1:36761/solr/collection1_shard2_replica_t6/] nUpdates=100
[junit4] 2> 1556006 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.PeerSync PeerSync: core=collection1_shard2_replica_t4 url=http://127.0.0.1:43489/solr DONE. We have no versions. sync failed.
[junit4] 2> 1556007 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.S.Request [collection1_shard2_replica_t6] webapp=/solr path=/get params={distrib=false&qt=/get&fingerprint=false&getVersions=100&wt=javabin&version=2} status=0 QTime=0
[junit4] 2> 1556007 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SyncStrategy Leader's attempt to sync with shard failed, moving to the next candidate
[junit4] 2> 1556007 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
[junit4] 2> 1556007 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext Found all replicas participating in election, clear LIR
[junit4] 2> 1556007 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController collection1_shard2_replica_t4 stopping background replication from leader
[junit4] 2> 1556008 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/ shard2
[junit4] 2> 1556109 INFO (zkCallback-4856-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1556109 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkController I am the leader, no recovery necessary
[junit4] 2> 1556110 INFO (qtp1734158036-19690) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node7&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t4&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=1738
[junit4] 2> 1556211 INFO (zkCallback-4856-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1556211 INFO (zkCallback-4870-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1556512 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ZkController collection1_shard1_replica_t1 starting background replication from leader
[junit4] 2> 1556512 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 1556514 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 1556514 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1556515 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node3&collection.configName=config&newCollection=true&name=collection1_shard1_replica_t1&action=CREATE&numShards=2&collection=collection1&shard=shard1&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2143
[junit4] 2> 1556521 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ZkController collection1_shard2_replica_t6 starting background replication from leader
[junit4] 2> 1556521 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.c.ReplicateFromLeader Will start replication from leader with poll interval: 00:00:03
[junit4] 2> 1556522 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Poll scheduled at an interval of 3000ms
[junit4] 2> 1556522 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
[junit4] 2> 1556522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/
[junit4] 2> 1556522 INFO (qtp1734158036-19700) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1556522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1556522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1556522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1556522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1556522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1556523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@76525a78[collection1_shard2_replica_t6] main]
[junit4] 2> 1556557 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&coreNodeName=core_node8&collection.configName=config&newCollection=true&name=collection1_shard2_replica_t6&action=CREATE&numShards=2&collection=collection1&shard=shard2&wt=javabin&version=2&replicaType=TLOG} status=0 QTime=2183
[junit4] 2> 1556559 INFO (searcherExecutor-6821-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@76525a78[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1556559 INFO (qtp1575998988-19701) [n:127.0.0.1:36761_solr ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 30 seconds. Check all shard replicas
[junit4] 2> 1556623 INFO (zkCallback-4856-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1556623 INFO (zkCallback-4870-thread-2) [ ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/collection1/state.json] for collection [collection1] has occurred - updating... (live nodes size: [2])
[junit4] 2> 1557559 INFO (qtp1575998988-19701) [n:127.0.0.1:36761_solr ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={pullReplicas=0&collection.configName=config&maxShardsPerNode=2&name=collection1&nrtReplicas=0&action=CREATE&numShards=2&tlogReplicas=2&wt=javabin&version=2} status=0 QTime=3497
[junit4] 2> 1557559 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4] 2> 1557560 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.AbstractDistribZkTestBase Recoveries finished - collection: collection1
[junit4] 2> 1557591 INFO (TEST-TestCloudRecovery.leaderRecoverFromLogOnStartupTest-seed#[75CA6A86654D6364]) [ ] o.a.s.SolrTestCaseJ4 ###Starting leaderRecoverFromLogOnStartupTest
[junit4] 2> 1557597 INFO (qtp1734158036-19688) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t1] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1612200889196478464&distrib.from=http://127.0.0.1:36761/solr/collection1_shard1_replica_t2/&wt=javabin&version=2}{deleteByQuery=*:* (-1612200889196478464)} 0 0
[junit4] 2> 1557597 INFO (qtp1575998988-19689) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard1 to Terms{values={core_node3=1, core_node5=1}, version=2}
[junit4] 2> 1557597 INFO (qtp1575998988-19689) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:* (-1612200889196478464)} 0 3
[junit4] 2> 1557598 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t6] webapp=/solr path=/update params={update.distrib=FROMLEADER&_version_=-1612200889195429888&distrib.from=http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/&wt=javabin&version=2}{deleteByQuery=*:* (-1612200889195429888)} 0 0
[junit4] 2> 1557599 INFO (qtp1734158036-19687) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.ZkShardTerms Successful update of terms at /collections/collection1/terms/shard2 to Terms{values={core_node7=1, core_node8=1}, version=2}
[junit4] 2> 1557599 INFO (qtp1734158036-19687) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={wt=javabin&version=2}{deleteByQuery=*:* (-1612200889195429888)} 0 7
[junit4] 2> 1557601 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1612200889204867072,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1557601 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 start commit{_version_=1612200889204867072,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4] 2> 1557601 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1557601 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 No uncommitted changes. Skipping IW.commit.
[junit4] 2> 1557601 INFO (qtp1734158036-19698) [n:127.0.0.1:43489_solr c:collection1 s:shard1 r:core_node3 x:collection1_shard1_replica_t1] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 1557601 INFO (qtp1575998988-19699) [n:127.0.0.1:36761_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.u.TestInjection Start waiting for replica in sync with leader
[junit4] 2> 1557601 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1557601 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard2_replica_t4] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 1557601 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.DirectUpdateHandler2 end_commit_flush
[junit4] 2> 1557601 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.u.p.LogUpdateProcessorFactory [collection1_shard1_replica_t2] webapp=/solr path=/update params={update.distrib=FROMLEADER&waitSearcher=true&openSearcher=true&commit=true&softCommit=false&distrib.from=http://127.0.0.1:43489/solr/collection1_shard2_replica_t4/&commit_end_point=true&wt=javabin&version=2&expungeDeletes=false}{commit=} 0 0
[junit4] 2> 1557602 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1557602 INFO (qtp1575998988-19697) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1558063 INFO (OverseerCollectionConfigSetProcessor-72261427679002628-127.0.0.1:36761_solr-n_0000000000) [n:127.0.0.1:36761_solr ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist. Requestor may have disconnected from ZooKeeper
[junit4] 2> 1558104 INFO (qtp1734158036-19772) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1558105 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1558606 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1558606 INFO (qtp1734158036-19692) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1559108 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1559108 INFO (qtp1734158036-19688) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1559289 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Updated masterUrl to http://127.0.0.1:36761/solr/collection1_shard1_replica_t2/
[junit4] 2> 1559291 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1559291 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1559291 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1559291 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1559291 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1559291 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1559292 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@90582e0[collection1_shard1_replica_t1] main]
[junit4] 2> 1559293 INFO (searcherExecutor-6822-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@90582e0[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1559523 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1559523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1559523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1559523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1559523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1559523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1559524 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@6277f0cf[collection1_shard2_replica_t6] main]
[junit4] 2> 1559524 INFO (searcherExecutor-6821-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@6277f0cf[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1559610 INFO (qtp1575998988-19697) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1559610 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1560112 INFO (qtp1734158036-19772) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1560112 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1560613 INFO (qtp1734158036-19692) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1560613 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1561115 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1561116 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1561617 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1561617 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1562119 INFO (qtp1734158036-19688) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1562119 INFO (qtp1575998988-19697) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1562289 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1562290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1562290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1562290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1562290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1562290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1562290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5c0d0c98[collection1_shard1_replica_t1] main]
[junit4] 2> 1562291 INFO (searcherExecutor-6822-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@5c0d0c98[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1562522 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1562522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1562522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1562522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1562522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1562522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1562523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5740af0c[collection1_shard2_replica_t6] main]
[junit4] 2> 1562524 INFO (searcherExecutor-6821-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@5740af0c[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1562621 INFO (qtp1734158036-19772) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1562621 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1563122 INFO (qtp1734158036-19688) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1563122 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1563624 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1563624 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1564126 INFO (qtp1734158036-19692) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1564126 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1564627 INFO (qtp1575998988-19697) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1564627 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1565129 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1565129 INFO (qtp1734158036-19688) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1565289 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1565290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1565290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1565290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1565290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1565290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1565290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@28177794[collection1_shard1_replica_t1] main]
[junit4] 2> 1565291 INFO (searcherExecutor-6822-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@28177794[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1565523 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1565523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1565523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1565523 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1565524 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1565524 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1565524 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@1647312[collection1_shard2_replica_t6] main]
[junit4] 2> 1565525 INFO (searcherExecutor-6821-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@1647312[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1565631 INFO (qtp1734158036-19772) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1565631 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1566133 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1566133 INFO (qtp1734158036-19692) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1566635 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1566635 INFO (qtp1575998988-19691) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1567136 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1567136 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1567638 INFO (qtp1575998988-19686) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1567639 INFO (qtp1734158036-19772) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1568140 INFO (qtp1575998988-19775) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1568140 INFO (qtp1734158036-19692) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1568289 INFO (qtp1575998988-19693) [n:127.0.0.1:36761_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t2] o.a.s.c.S.Request [collection1_shard1_replica_t2] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1568290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1568290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1568290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1568290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1568290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1568290 INFO (indexFetcher-6841-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@3fe1d085[collection1_shard1_replica_t1] main]
[junit4] 2> 1568291 INFO (searcherExecutor-6822-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard1_replica_t1] Registered new searcher Searcher@3fe1d085[collection1_shard1_replica_t1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1568522 INFO (qtp1734158036-19696) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=indexversion} status=0 QTime=0
[junit4] 2> 1568522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's generation: 1
[junit4] 2> 1568522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Master's version: 0
[junit4] 2> 1568522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's generation: 1
[junit4] 2> 1568522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher Slave's version: 0
[junit4] 2> 1568522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.h.IndexFetcher New index in Master. Deleting mine...
[junit4] 2> 1568522 INFO (indexFetcher-6843-thread-1) [ ] o.a.s.s.SolrIndexSearcher Opening [Searcher@5b406f[collection1_shard2_replica_t6] main]
[junit4] 2> 1568523 INFO (searcherExecutor-6821-thread-1) [ ] o.a.s.c.SolrCore [collection1_shard2_replica_t6] Registered new searcher Searcher@5b406f[collection1_shard2_replica_t6] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
[junit4] 2> 1568641 INFO (qtp1734158036-19785) [n:127.0.0.1:43489_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.S.Request [collection1_shard2_replica_t4] webapp=/solr path=/replication params={qt=/replication&wt=javabin&version=2&command=details} status=0 QTime=0
[junit4] 2> 1568641 INFO (qtp1575998988-19697) [n:127.0.0.1:36761_solr c:collec
[...truncated too long message...]
Executor-320-thread-1) [n:127.0.0.1:34535_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.c.SolrCore [collection1_shard1_replica_t1] CLOSING SolrCore org.apache.solr.core.SolrCore@7e33cd98
[junit4] 2> 108449 INFO (coreCloseExecutor-319-thread-2) [n:127.0.0.1:33881_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2a59d0f8: rootName = solr_33881, domain = solr.core.collection1.shard2.replica_t6, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_t6 / com.codahale.metrics.MetricRegistry@6cc096a2
[junit4] 2> 108449 INFO (coreCloseExecutor-320-thread-1) [n:127.0.0.1:34535_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard1.replica_t1, tag=7e33cd98
[junit4] 2> 108450 INFO (coreCloseExecutor-320-thread-1) [n:127.0.0.1:34535_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@3103241c: rootName = solr_34535, domain = solr.core.collection1.shard1.replica_t1, service url = null, agent id = null] for registry solr.core.collection1.shard1.replica_t1 / com.codahale.metrics.MetricRegistry@16efafea
[junit4] 2> 108450 INFO (coreCloseExecutor-320-thread-2) [n:127.0.0.1:34535_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.c.SolrCore [collection1_shard2_replica_t4] CLOSING SolrCore org.apache.solr.core.SolrCore@2ba40bbf
[junit4] 2> 108457 INFO (coreCloseExecutor-320-thread-2) [n:127.0.0.1:34535_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.core.collection1.shard2.replica_t4, tag=2ba40bbf
[junit4] 2> 108457 INFO (coreCloseExecutor-320-thread-2) [n:127.0.0.1:34535_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.r.SolrJmxReporter Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@40f9e6a3: rootName = solr_34535, domain = solr.core.collection1.shard2.replica_t4, service url = null, agent id = null] for registry solr.core.collection1.shard2.replica_t4 / com.codahale.metrics.MetricRegistry@33821e29
[junit4] 2> 108460 INFO (coreCloseExecutor-319-thread-1) [n:127.0.0.1:33881_solr c:collection1 s:shard1 r:core_node5 x:collection1_shard1_replica_t3] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=22ee23d8
[junit4] 2> 108460 INFO (coreCloseExecutor-319-thread-2) [n:127.0.0.1:33881_solr c:collection1 s:shard2 r:core_node8 x:collection1_shard2_replica_t6] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=5ff94c76
[junit4] 2> 108465 INFO (jetty-closer-113-thread-1) [ ] o.a.s.c.Overseer Overseer (id=72261580763955221-127.0.0.1:33881_solr-n_0000000004) closing
[junit4] 2> 108465 INFO (OverseerStateUpdate-72261580763955221-127.0.0.1:33881_solr-n_0000000004) [n:127.0.0.1:33881_solr ] o.a.s.c.Overseer Overseer Loop exiting : 127.0.0.1:33881_solr
[junit4] 2> 108465 WARN (OverseerAutoScalingTriggerThread-72261580763955221-127.0.0.1:33881_solr-n_0000000004) [ ] o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread woken up but we are closed, exiting.
[junit4] 2> 108468 INFO (zkCallback-211-thread-3) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:34535_solr
[junit4] 2> 108468 INFO (jetty-closer-113-thread-1) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@2a2d4c37{/solr,null,UNAVAILABLE}
[junit4] 2> 108468 INFO (jetty-closer-113-thread-1) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 108469 INFO (coreCloseExecutor-320-thread-1) [n:127.0.0.1:34535_solr c:collection1 s:shard1 r:core_node2 x:collection1_shard1_replica_t1] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard1.leader, tag=7e33cd98
[junit4] 2> 108469 INFO (coreCloseExecutor-320-thread-2) [n:127.0.0.1:34535_solr c:collection1 s:shard2 r:core_node7 x:collection1_shard2_replica_t4] o.a.s.m.SolrMetricManager Closing metric reporters for registry=solr.collection.collection1.shard2.leader, tag=2ba40bbf
[junit4] 2> 108473 INFO (jetty-closer-113-thread-2) [ ] o.e.j.s.h.ContextHandler Stopped o.e.j.s.ServletContextHandler@30de6f5c{/solr,null,UNAVAILABLE}
[junit4] 2> 108473 INFO (jetty-closer-113-thread-2) [ ] o.e.j.s.session node0 Stopped scavenging
[junit4] 2> 108473 ERROR (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.z.s.ZooKeeperServer ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes
[junit4] 2> 108473 INFO (SUITE-TestCloudRecovery-seed#[75CA6A86654D6364]-worker) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46767 46767
[junit4] 2> 109443 INFO (Thread-79) [ ] o.a.s.c.ZkTestServer connecting to 127.0.0.1:46767 46767
[junit4] 2> 109443 WARN (Thread-79) [ ] o.a.s.c.ZkTestServer Watch limit violations:
[junit4] 2> Maximum concurrent create/delete watches above limit:
[junit4] 2>
[junit4] 2> 20 /solr/collections/collection1/terms/shard1
[junit4] 2> 19 /solr/collections/collection1/terms/shard2
[junit4] 2> 13 /solr/aliases.json
[junit4] 2> 13 /solr/clusterprops.json
[junit4] 2> 6 /solr/security.json
[junit4] 2> 6 /solr/configs/config
[junit4] 2>
[junit4] 2> Maximum concurrent data watches above limit:
[junit4] 2>
[junit4] 2> 14 /solr/collections/collection1/state.json
[junit4] 2> 13 /solr/clusterstate.json
[junit4] 2> 5 /solr/autoscaling.json
[junit4] 2>
[junit4] 2> Maximum concurrent children watches above limit:
[junit4] 2>
[junit4] 2> 13 /solr/live_nodes
[junit4] 2> 13 /solr/collections
[junit4] 2> 5 /solr/overseer/queue
[junit4] 2> 5 /solr/autoscaling/events/.scheduled_maintenance
[junit4] 2> 5 /solr/autoscaling/events/.auto_add_replicas
[junit4] 2> 5 /solr/overseer/queue-work
[junit4] 2> 5 /solr/overseer/collection-queue-work
[junit4] 2>
[junit4] 2> NOTE: leaving temporary files on disk at: /home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestCloudRecovery_75CA6A86654D6364-001
[junit4] 2> NOTE: test params are: codec=Asserting(Lucene80): {id=PostingsFormat(name=LuceneVarGapFixedInterval)}, docValues:{_version_=DocValuesFormat(name=Asserting)}, maxPointsInLeafNode=430, maxMBSortInHeap=7.184585892855316, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@1a7dc50d), locale=ar-OM, timezone=US/Arizona
[junit4] 2> NOTE: Linux 4.15.0-32-generic amd64/Oracle Corporation 1.8.0_172 (64-bit)/cpus=8,threads=1,free=249901632,total=518979584
[junit4] 2> NOTE: All tests run in this JVM: [TestCloudRecovery, TestCloudRecovery]
[junit4] Completed [4/5 (1!)] on J0 in 51.73s, 2 tests, 1 failure <<< FAILURES!
[junit4]
[junit4] Suite: org.apache.solr.cloud.TestCloudRecovery
[junit4] OK 36.4s J2 | TestCloudRecovery.leaderRecoverFromLogOnStartupTest
[junit4] OK 9.78s J2 | TestCloudRecovery.corruptedLogTest
[junit4] Completed [5/5 (1!)] on J2 in 50.77s, 2 tests
[junit4]
[junit4]
[junit4] Tests with failures [seed: 75CA6A86654D6364]:
[junit4] - org.apache.solr.cloud.TestCloudRecovery.corruptedLogTest
[junit4]
[junit4]
[junit4] JVM J0: 0.41 .. 110.03 = 109.62s
[junit4] JVM J1: 0.41 .. 61.02 = 60.61s
[junit4] JVM J2: 0.41 .. 111.81 = 111.40s
[junit4] Execution time total: 1 minute 51 seconds
[junit4] Tests summary: 5 suites, 10 tests, 1 failure
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1567: The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1092: There were test failures: 5 suites, 10 tests, 1 failure [seed: 75CA6A86654D6364]
Total time: 1 minute 53 seconds
[repro] Setting last failure code to 256
[repro] Failures:
[repro] 1/5 failed: org.apache.solr.cloud.TestCloudRecovery
[repro] Exiting with code 256
+ mv lucene/build lucene/build.repro
+ mv solr/build solr/build.repro
+ mv lucene/build.orig lucene/build
+ mv solr/build.orig solr/build
Archiving artifacts
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Parsing warnings in console log with parser Java Compiler (javac)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
<Git Blamer> Using GitBlamer to create author and commit information for all warnings.
<Git Blamer> GIT_COMMIT=dd088fb83eeb48c752cc69a5dd173aa1c5224bc9, workspace=/var/lib/jenkins/workspace/Lucene-Solr-master-Linux
[WARNINGS] Computing warning deltas based on reference build #22895
Recording test results
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Build step 'Publish JUnit test result report' changed build result to UNSTABLE
Email was triggered for: Unstable (Test Failures)
Sending email for trigger: Unstable (Test Failures)
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2