You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@lucene.apache.org by Policeman Jenkins Server <je...@thetaphi.de> on 2019/12/23 01:05:20 UTC

[JENKINS] Lucene-Solr-8.x-Solaris (64bit/jdk1.8.0) - Build # 468 - Unstable!

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/468/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.OverseerTaskQueueTest.testDistributedQueue

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
	at __randomizedtesting.SeedInfo.seed([729C3DF8DF48C9FB]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.OverseerTaskQueueTest

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
	at __randomizedtesting.SeedInfo.seed([729C3DF8DF48C9FB]:0)




Build Log:
[...truncated 16393 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTaskQueueTest
   [junit4]   2> 2958932 INFO  (SUITE-OverseerTaskQueueTest-seed#[729C3DF8DF48C9FB]-worker) [     ] o.a.s.SolrTestCaseJ4 Created dataDir: /export/home/jenkins/workspace/Lucene-Solr-8.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerTaskQueueTest_729C3DF8DF48C9FB-001/data-dir-287-001
   [junit4]   2> 2958932 WARN  (SUITE-OverseerTaskQueueTest-seed#[729C3DF8DF48C9FB]-worker) [     ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=30 numCloses=30
   [junit4]   2> 2958932 INFO  (SUITE-OverseerTaskQueueTest-seed#[729C3DF8DF48C9FB]-worker) [     ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2958934 INFO  (SUITE-OverseerTaskQueueTest-seed#[729C3DF8DF48C9FB]-worker) [     ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN)
   [junit4]   2> 2958934 INFO  (SUITE-OverseerTaskQueueTest-seed#[729C3DF8DF48C9FB]-worker) [     ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2958939 INFO  (TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueue
   [junit4]   2> 2958940 INFO  (TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]) [     ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2958940 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2958941 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 2959041 INFO  (TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]) [     ] o.a.s.c.ZkTestServer start zk server on port:49018
   [junit4]   2> 2959041 INFO  (TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]) [     ] o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:49018
   [junit4]   2> 2959041 INFO  (TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:49018
   [junit4]   2> 2959041 INFO  (TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 49018
   [junit4]   2> ??? 23, 2019 2:27:51 ? com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: org.apache.solr.cloud.OverseerTaskQueueTest
   [junit4]   2> ==== jstack at approximately timeout time ====
   [junit4]   2> "ProcessThread(sid:0 cport:49018):" ID=45933 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3614fdb9
   [junit4]   2> 	at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3614fdb9
   [junit4]   2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:134)
   [junit4]   2> 
   [junit4]   2> "SyncThread:0" ID=45932 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2e836eeb
   [junit4]   2> 	at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2e836eeb
   [junit4]   2> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2> 	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:109)
   [junit4]   2> 
   [junit4]   2> "SessionTracker" ID=45931 TIMED_WAITING
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
   [junit4]   2> 
   [junit4]   2> "ConnnectionExpirer" ID=45928 TIMED_WAITING
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:574)
   [junit4]   2> 
   [junit4]   2> "NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0" ID=45930 RUNNABLE (in native code)
   [junit4]   2> 	at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]   2> 	at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]   2> 	at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]   2> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	- locked sun.nio.ch.Util$3@67f623fa
   [junit4]   2> 	- locked java.util.Collections$UnmodifiableSet@112e3878
   [junit4]   2> 	- locked sun.nio.ch.DevPollSelectorImpl@46e6d1
   [junit4]   2> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:229)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:205)
   [junit4]   2> 
   [junit4]   2> "NIOServerCxnFactory.SelectorThread-0" ID=45929 RUNNABLE (in native code)
   [junit4]   2> 	at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]   2> 	at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]   2> 	at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]   2> 	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2> 	- locked sun.nio.ch.Util$3@35872404
   [junit4]   2> 	- locked java.util.Collections$UnmodifiableSet@63b838bd
   [junit4]   2> 	- locked sun.nio.ch.DevPollSelectorImpl@3b5314a
   [junit4]   2> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2> 	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:426)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:391)
   [junit4]   2> 
   [junit4]   2> "ZkTestServer Run Thread" ID=45927 WAITING on org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@34f22f83
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread@34f22f83
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:950)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:343)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:566)
   [junit4]   2> 
   [junit4]   2> "TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB]" ID=45926 RUNNABLE (in native code)
   [junit4]   2> 	at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2> 	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2> 	at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2> 	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
   [junit4]   2> 	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
   [junit4]   2> 	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
   [junit4]   2> 	- locked java.io.InputStreamReader@38598a1d
   [junit4]   2> 	at java.io.InputStreamReader.read(InputStreamReader.java:184)
   [junit4]   2> 	at java.io.BufferedReader.fill(BufferedReader.java:161)
   [junit4]   2> 	at java.io.BufferedReader.readLine(BufferedReader.java:324)
   [junit4]   2> 	- locked java.io.InputStreamReader@38598a1d
   [junit4]   2> 	at java.io.BufferedReader.readLine(BufferedReader.java:389)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer.send4LetterWord(ZkTestServer.java:721)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer.waitForServerUp(ZkTestServer.java:669)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:597)
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:526)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueueTest.setupZk(DistributedQueueTest.java:345)
   [junit4]   2> 	at org.apache.solr.cloud.DistributedQueueTest.setUp(DistributedQueueTest.java:50)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2> 	at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2> 	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2> 	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> "SUITE-OverseerTaskQueueTest-seed#[729C3DF8DF48C9FB]" ID=45925 RUNNABLE
   [junit4]   2> 	at sun.management.ThreadImpl.dumpThreads0(Native Method)
   [junit4]   2> 	at sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:454)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.formatThreadStacksFull(ThreadLeakControl.java:681)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl.access$1000(ThreadLeakControl.java:65)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.ThreadLeakControl$2.evaluate(ThreadLeakControl.java:415)
   [junit4]   2> 	- locked java.lang.Object@1a06207b
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:708)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.access$200(RandomizedRunner.java:138)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner$2.run(RandomizedRunner.java:629)
   [junit4]   2> 
   [junit4]   2> "org.apache.hadoop.hdfs.PeerCache@10a9a1fe" ID=10731 TIMED_WAITING
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:253)
   [junit4]   2> 	at org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:46)
   [junit4]   2> 	at org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:124)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> "org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" ID=10663 WAITING on java.lang.ref.ReferenceQueue$Lock@30937d10
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@30937d10
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
   [junit4]   2> 	at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:3762)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> "Log4j2-TF-1-AsyncLoggerConfig-1" ID=11 TIMED_WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@15b8d2b6
   [junit4]   2> 	at sun.misc.Unsafe.park(Native Method)
   [junit4]   2> 	- timed waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@15b8d2b6
   [junit4]   2> 	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:748)
   [junit4]   2> 
   [junit4]   2> "JUnit4-serializer-daemon" ID=8 TIMED_WAITING
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.events.Serializer$1.run(Serializer.java:50)
   [junit4]   2> 
   [junit4]   2> "Signal Dispatcher" ID=4 RUNNABLE
   [junit4]   2> 
   [junit4]   2> "Finalizer" ID=3 WAITING on java.lang.ref.ReferenceQueue$Lock@6145bb7e
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.ReferenceQueue$Lock@6145bb7e
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
   [junit4]   2> 	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
   [junit4]   2> 	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
   [junit4]   2> 
   [junit4]   2> "Reference Handler" ID=2 WAITING on java.lang.ref.Reference$Lock@4a6b5ad5
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on java.lang.ref.Reference$Lock@4a6b5ad5
   [junit4]   2> 	at java.lang.Object.wait(Object.java:502)
   [junit4]   2> 	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   [junit4]   2> 	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
   [junit4]   2> 
   [junit4]   2> "main" ID=1 WAITING on com.carrotsearch.randomizedtesting.RandomizedRunner$2@46e4d68e
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> 	- waiting on com.carrotsearch.randomizedtesting.RandomizedRunner$2@46e4d68e
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSuite(RandomizedRunner.java:639)
   [junit4]   2> 	at com.carrotsearch.randomizedtesting.RandomizedRunner.run(RandomizedRunner.java:496)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.execute(SlaveMain.java:269)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMain.main(SlaveMain.java:394)
   [junit4]   2> 	at com.carrotsearch.ant.tasks.junit4.slave.SlaveMainSafe.main(SlaveMainSafe.java:13)
   [junit4]   2> 
   [junit4]   2> ^^==============================================
   [junit4]   2> 
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTaskQueueTest -Dtests.method=testDistributedQueue -Dtests.seed=729C3DF8DF48C9FB -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=MIT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   7200s J1 | OverseerTaskQueueTest.testDistributedQueue <<<
   [junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([729C3DF8DF48C9FB]:0)
   [junit4]   2> ??? 23, 2019 2:27:51 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 8 leaked thread(s).
   [junit4]   2> ??? 23, 2019 2:28:01 ? com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 8 threads leaked from SUITE scope at org.apache.solr.cloud.OverseerTaskQueueTest: 
   [junit4]   2>    1) Thread[id=45926, name=TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB], state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2>         at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2>         at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2>         at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2>         at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
   [junit4]   2>         at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
   [junit4]   2>         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
   [junit4]   2>         at java.io.InputStreamReader.read(InputStreamReader.java:184)
   [junit4]   2>         at java.io.BufferedReader.fill(BufferedReader.java:161)
   [junit4]   2>         at java.io.BufferedReader.readLine(BufferedReader.java:324)
   [junit4]   2>         at java.io.BufferedReader.readLine(BufferedReader.java:389)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.send4LetterWord(ZkTestServer.java:721)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.waitForServerUp(ZkTestServer.java:669)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:597)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:526)
   [junit4]   2>         at org.apache.solr.cloud.DistributedQueueTest.setupZk(DistributedQueueTest.java:345)
   [junit4]   2>         at org.apache.solr.cloud.DistributedQueueTest.setUp(DistributedQueueTest.java:50)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    2) Thread[id=45927, name=ZkTestServer Run Thread, state=WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at java.lang.Object.wait(Native Method)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1252)
   [junit4]   2>         at java.lang.Thread.join(Thread.java:1326)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:950)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:343)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:566)
   [junit4]   2>    3) Thread[id=45930, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:229)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:205)
   [junit4]   2>    4) Thread[id=45933, name=ProcessThread(sid:0 cport:49018):, state=WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:134)
   [junit4]   2>    5) Thread[id=45929, name=NIOServerCxnFactory.SelectorThread-0, state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll0(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.poll(DevPollArrayWrapper.java:223)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:98)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:426)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:391)
   [junit4]   2>    6) Thread[id=45928, name=ConnnectionExpirer, state=TIMED_WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$ConnectionExpirerThread.run(NIOServerCnxnFactory.java:574)
   [junit4]   2>    7) Thread[id=45932, name=SyncThread:0, state=WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
   [junit4]   2>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
   [junit4]   2>         at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
   [junit4]   2>         at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:109)
   [junit4]   2>    8) Thread[id=45931, name=SessionTracker, state=TIMED_WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at java.lang.Thread.sleep(Native Method)
   [junit4]   2>         at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151)
   [junit4]   2> ??? 23, 2019 2:28:01 ? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=45926, name=TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB], state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    2) Thread[id=45927, name=ZkTestServer Run Thread, state=WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    3) Thread[id=45930, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    4) Thread[id=45933, name=ProcessThread(sid:0 cport:49018):, state=WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    5) Thread[id=45929, name=NIOServerCxnFactory.SelectorThread-0, state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    6) Thread[id=45928, name=ConnnectionExpirer, state=TIMED_WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    7) Thread[id=45932, name=SyncThread:0, state=WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>    8) Thread[id=45931, name=SessionTracker, state=TIMED_WAITING, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2> 10169188 WARN  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Server interrupted
   [junit4]   2>           => java.lang.InterruptedException
   [junit4]   2> 	at java.lang.Object.wait(Native Method)
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.lang.Object.wait(Native Method) ~[?:1.8.0_221]
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1252) ~[?:1.8.0_221]
   [junit4]   2> 	at java.lang.Thread.join(Thread.java:1326) [?:1.8.0_221]
   [junit4]   2> 	at org.apache.zookeeper.server.NIOServerCnxnFactory.join(NIOServerCnxnFactory.java:950) ~[zookeeper-3.5.5.jar:3.5.5]
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$ZKServerMain.runFromConfig(ZkTestServer.java:343) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.ZkTestServer$2.run(ZkTestServer.java:566) ~[java/:?]
   [junit4]   2> 10169188 ERROR (SyncThread:0) [     ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : SyncThread:0
   [junit4]   2>           => java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_221]
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_221]
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[?:1.8.0_221]
   [junit4]   2> 	at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:109) ~[zookeeper-3.5.5.jar:3.5.5]
   [junit4]   2> 10169188 ERROR (ProcessThread(sid:0 cport:49018):) [     ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : ProcessThread(sid:0 cport:49018):
   [junit4]   2>           => java.lang.InterruptedException
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
   [junit4]   2> java.lang.InterruptedException: null
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) ~[?:1.8.0_221]
   [junit4]   2> 	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) ~[?:1.8.0_221]
   [junit4]   2> 	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) ~[?:1.8.0_221]
   [junit4]   2> 	at org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:134) ~[zookeeper-3.5.5.jar:3.5.5]
   [junit4]   2> 10169189 ERROR (SessionTracker) [     ] o.a.z.s.ZooKeeperCriticalThread Severe unrecoverable error, from thread : SessionTracker
   [junit4]   2>           => java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method)
   [junit4]   2> java.lang.InterruptedException: sleep interrupted
   [junit4]   2> 	at java.lang.Thread.sleep(Native Method) [?:1.8.0_221]
   [junit4]   2> 	at org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:151) ~[zookeeper-3.5.5.jar:3.5.5]
   [junit4]   2> ??? 23, 2019 2:28:04 ? com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=45926, name=TEST-OverseerTaskQueueTest.testDistributedQueue-seed#[729C3DF8DF48C9FB], state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at java.net.SocketInputStream.socketRead0(Native Method)
   [junit4]   2>         at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
   [junit4]   2>         at java.net.SocketInputStream.read(SocketInputStream.java:171)
   [junit4]   2>         at java.net.SocketInputStream.read(SocketInputStream.java:141)
   [junit4]   2>         at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
   [junit4]   2>         at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
   [junit4]   2>         at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
   [junit4]   2>         at java.io.InputStreamReader.read(InputStreamReader.java:184)
   [junit4]   2>         at java.io.BufferedReader.fill(BufferedReader.java:161)
   [junit4]   2>         at java.io.BufferedReader.readLine(BufferedReader.java:324)
   [junit4]   2>         at java.io.BufferedReader.readLine(BufferedReader.java:389)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.send4LetterWord(ZkTestServer.java:721)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.waitForServerUp(ZkTestServer.java:669)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:597)
   [junit4]   2>         at org.apache.solr.cloud.ZkTestServer.run(ZkTestServer.java:526)
   [junit4]   2>         at org.apache.solr.cloud.DistributedQueueTest.setupZk(DistributedQueueTest.java:345)
   [junit4]   2>         at org.apache.solr.cloud.DistributedQueueTest.setUp(DistributedQueueTest.java:50)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   [junit4]   2>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>         at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:972)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
   [junit4]   2>         at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
   [junit4]   2>         at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
   [junit4]   2>         at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
   [junit4]   2>         at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
   [junit4]   2>         at java.lang.Thread.run(Thread.java:748)
   [junit4]   2>    2) Thread[id=45930, name=NIOServerCxnFactory.AcceptThread:0.0.0.0/0.0.0.0:0, state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.interrupt(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollArrayWrapper.interrupt(DevPollArrayWrapper.java:300)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.wakeup(DevPollSelectorImpl.java:207)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:213)
   [junit4]   2>         at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:219)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.select(NIOServerCnxnFactory.java:229)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$AcceptThread.run(NIOServerCnxnFactory.java:205)
   [junit4]   2>    3) Thread[id=45929, name=NIOServerCxnFactory.SelectorThread-0, state=RUNNABLE, group=TGRP-OverseerTaskQueueTest]
   [junit4]   2>         at sun.nio.ch.IOUtil.drain(Native Method)
   [junit4]   2>         at sun.nio.ch.DevPollSelectorImpl.doSelect(DevPollSelectorImpl.java:109)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   [junit4]   2>         at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.select(NIOServerCnxnFactory.java:426)
   [junit4]   2>         at org.apache.zookeeper.server.NIOServerCnxnFactory$SelectorThread.run(NIOServerCnxnFactory.java:391)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=359, maxMBSortInHeap=6.405197713937243, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6ed6c961), locale=ar-IQ, timezone=MIT
   [junit4]   2> NOTE: SunOS 5.11 amd64/Oracle Corporation 1.8.0_221 (64-bit)/cpus=3,threads=4,free=242521072,total=503316480
   [junit4]   2> NOTE: All tests run in this JVM: [AssignTest, SolrCloudExampleTest, ScheduledMaintenanceTriggerTest, TestReqParamsAPI, TestDistributedMap, TestExportTool, SolrMetricsIntegrationTest, HttpPartitionTest, UtilsToolTest, TokenizerChainTest, TestJsonFacetRefinement, TestPKIAuthenticationPlugin, TestDynamicFieldCollectionResource, ShowFileRequestHandlerTest, TestRecovery, CurrencyFieldTypeTest, TestReRankQParserPlugin, TestSolrCachePerf, TestRTimerTree, TestImpersonationWithHadoopAuth, SolrShardReporterTest, TestPackages, TestFieldCacheVsDocValues, TestMissingGroups, TestRandomCollapseQParserPlugin, TestExpandComponent, TestMergePolicyConfig, IndexSizeTriggerSizeEstimationTest, TestMacros, HdfsWriteToMultipleCollectionsTest, RequiredFieldsTest, TestLRUStatsCache, TestMaxScoreQueryParser, FullHLLTest, TestSchemaSimilarityResource, TestDeleteCollectionOnDownNodes, TestDefaultStatsCache, TestDynamicFieldNamesIndexCorrectly, TestAnalyzedSuggestions, SimpleFacetsTest, TermVectorComponentDistributedTest, TestStandardQParsers, ExecutePlanActionTest, TestCustomSort, OverseerStatusTest, OverriddenZkACLAndCredentialsProvidersTest, SolrTestCaseJ4Test, TestStressUserVersions, TestSimComputePlanAction, TestMaxTokenLenTokenizer, RootFieldTest, TestLeaderElectionZkExpiry, NodeLostTriggerIntegrationTest, TestEmbeddedSolrServerAdminHandler, TestDistributedTracing, TestElisionMultitermQuery, CdcrBidirectionalTest, TestXIncludeConfig, TestExactSharedStatsCache, PreAnalyzedFieldTest, TestPseudoReturnFields, ShardRoutingTest, BasicFunctionalityTest, IgnoreCommitOptimizeUpdateProcessorFactoryTest, ImplicitSnitchTest, ZkFailoverTest, TestSearchPerf, DistributedFacetPivotSmallTest, LeaderElectionContextKeyTest, MaxSizeAutoCommitTest, TestAddFieldRealTimeGet, TestHdfsUpdateLog, TestJsonFacetsStatsParsing, FastVectorHighlighterTest, TestCoreDiscovery, MultiDestinationAuditLoggerTest, SparseHLLTest, TestXmlQParserPlugin, TestInPlaceUpdatesStandalone, UniqFieldsUpdateProcessorFactoryTest, TestCollectionAPI, TestNamedUpdateProcessors, TestClassicSimilarityFactory, TestFuzzyAnalyzedSuggestions, TestStreamBody, TestFilteredDocIdSet, TestDocTermOrds, OverseerTest, DistributedMLTComponentTest, TestSystemIdResolver, TestUtils, TestSolrCloudSnapshots, TestSolrQueryParser, TestClusterStateMutator, TestHighlightDedupGrouping, TestTolerantUpdateProcessorCloud, BlockJoinFacetRandomTest, TestSimNodeLostTrigger, SolrLogAuditLoggerPluginTest, TestSolrFieldCacheBean, XMLAtomicUpdateMultivalueTest, RequestHandlersTest, HdfsCollectionsAPIDistributedZkTest, AuditLoggerPluginTest, QueryResultKeyTest, SolrCmdDistributorTest, SOLR749Test, TestFoldingMultitermQuery, TestFieldCache, TestFieldCollectionResource, LoggingHandlerTest, MoveReplicaHDFSFailoverTest, TestMultiValuedNumericRangeQuery, HdfsRecoveryZkTest, TestHashQParserPlugin, TestShortCircuitedRequests, RangeFacetCloudTest, IndexSizeEstimatorTest, SecurityConfHandlerTest, TestSimScenario, TestReload, AddSchemaFieldsUpdateProcessorFactoryTest, DistributedIntervalFacetingTest, CollectionStateFormat2Test, TestUtilizeNode, TestLegacyNumericUtils, CollectionsAPIAsyncDistributedZkTest, TestPrepRecovery, TestRandomDVFaceting, TestLegacyBM25SimilarityFactory, JavaBinAtomicUpdateMultivalueTest, CursorMarkTest, TestRawTransformer, TestLMDirichletSimilarityFactory, TestManagedStopFilterFactory, TestCloudInspectUtil, DirectSolrConnectionTest, TestHashPartitioner, TestFieldTypeCollectionResource, TestQuerySenderListener, TestStressCloudBlindAtomicUpdates, CircularListTest, SplitHandlerTest, SearchRateTriggerTest, DocValuesNotIndexedTest, ChangedSchemaMergeTest, TestHdfsBackupRestoreCore, TestPerFieldSimilarity, TestWriterPerf, CSVRequestHandlerTest, TestNestedUpdateProcessor, RoutingToNodesWithPropertiesTest, DeleteLastCustomShardedReplicaTest, TestConfig, TestNestedDocsSort, SliceStateTest, TestNumericRangeQuery64, TestFieldCacheSort, URLClassifyProcessorTest, TestConfigSetImmutable, CategoryRoutedAliasUpdateProcessorTest, TestOmitPositions, TestSolrCLIRunExample, TestSubQueryTransformerDistrib, TestLegacyField, MetricsHistoryHandlerTest, IndexBasedSpellCheckerTest, TransactionLogTest, HdfsNNFailoverTest, SolrSlf4jReporterTest, TestTrie, DistributedFacetPivotLongTailTest, TestSchemaNameResource, TestScoreJoinQPScore, ZkShardTermsTest, SpellCheckComponentTest, TestStressThreadBackup, ConnectionManagerTest, BlockDirectoryTest, RestartWhileUpdatingTest, TestGeoJSONResponseWriter, TestCloudPivotFacet, TestDynamicLoading, FileUtilsTest, TestUseDocValuesAsStored, PeerSyncReplicationTest, HealthCheckHandlerTest, TestSerializedLuceneMatchVersion, SimpleCollectionCreateDeleteTest, DimensionalRoutedAliasUpdateProcessorTest, AutoAddReplicasIntegrationTest, TestCloudRecovery, TestNoOpRegenerator, SmileWriterTest, TestCloudSearcherWarming, SpellingQueryConverterTest, SplitShardTest, TestQueryUtils, ChaosMonkeySafeLeaderTest, TestSubQueryTransformer, TestDFRSimilarityFactory, JWTAuthPluginIntegrationTest, TestDynamicURP, ReturnFieldsTest, TestStressLiveNodes, DateMathParserTest, CloneFieldUpdateProcessorFactoryTest, ConfigureRecoveryStrategyTest, TestRebalanceLeaders, TestSimpleQParserPlugin, DirectSolrSpellCheckerTest, V2ApiIntegrationTest, TestPullReplicaErrorHandling, LeaderElectionTest, PolyFieldTest, BlobRepositoryCloudTest, TestSha256AuthenticationProvider, HdfsTlogReplayBufferedWhileIndexingTest, DocValuesMultiTest, AnalysisAfterCoreReloadTest, ConvertedLegacyTest, CursorPagingTest, EchoParamsTest, MinimalSchemaTest, OutputWriterTest, SampleTest, SolrInfoBeanTest, TestCursorMarkWithoutUniqueKey, TestDistributedMissingSort, TestDocumentBuilder, TestGroupingSearch, TestJoin, TestSimpleTrackingShardHandler, TestSolrCoreProperties, TestSolrTestCaseJ4, TestTolerantSearch, PathHierarchyTokenizerFactoryTest, ProtectedTermFilterFactoryTest, TestCharFilters, TestDeprecatedFilters, TestLuceneMatchVersion, TestReversedWildcardFilterFactory, TestLuceneIndexBackCompat, TestEmbeddedSolrServerSchemaAPI, ConnectionReuseTest, AddReplicaTest, AssignBackwardCompatibilityTest, BasicDistributedZk2Test, BasicDistributedZkTest, ChaosMonkeyNothingIsSafeTest, ChaosMonkeyNothingIsSafeWithPullReplicasTest, ChaosMonkeySafeLeaderWithPullReplicasTest, CleanupOldIndexTest, CloudExitableDirectoryReaderTest, ClusterStateTest, ClusterStateUpdateTest, ConcurrentCreateRoutedAliasTest, CreateRoutedAliasTest, DeleteNodeTest, DeleteReplicaTest, DeleteShardTest, DeleteStatusTest, DistribCursorPagingTest, DistribDocExpirationUpdateProcessorTest, DistributedQueueTest, DistributedVersionInfoTest, ForceLeaderWithTlogReplicasTest, FullSolrCloudDistribCmdsTest, LeaderFailoverAfterPartitionTest, LeaderTragicEventTest, LeaderVoteWaitTimeoutTest, MigrateRouteKeyTest, MultiThreadedOCPTest, NestedShardedAtomicUpdateTest, OverseerTaskQueueTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OverseerTaskQueueTest -Dtests.seed=729C3DF8DF48C9FB -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=MIT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | OverseerTaskQueueTest (suite) <<<
   [junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([729C3DF8DF48C9FB]:0)
   [junit4] Completed [889/889 (1!)] on J1 in 7212.80s, 1 test, 2 errors <<< FAILURES!

[...truncated 52894 lines...]
[repro] Jenkins log URL: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/468/consoleText

[repro] Revision: f74a62f37e9ee156351cdf20c2549aeece0d2ce2

[repro] Ant options: "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC"
[repro] Repro line:  ant test  -Dtestcase=OverseerTaskQueueTest -Dtests.method=testDistributedQueue -Dtests.seed=729C3DF8DF48C9FB -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=MIT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] Repro line:  ant test  -Dtestcase=OverseerTaskQueueTest -Dtests.seed=729C3DF8DF48C9FB -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=MIT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[repro] JUnit rest result XML files will be moved to: ./repro-reports
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]    solr/core
[repro]       OverseerTaskQueueTest
[repro] ant compile-test

[...truncated 3622 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 -Dtests.class="*.OverseerTaskQueueTest" -Dtests.showOutput=onerror "-Dargs=-XX:-UseCompressedOops -XX:+UseParallelGC" -Dtests.seed=729C3DF8DF48C9FB -Dtests.slow=true -Dtests.locale=ar-IQ -Dtests.timezone=MIT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1

[...truncated 97 lines...]
[repro] Failures w/original seeds:
[repro]   0/5 failed: org.apache.solr.cloud.OverseerTaskQueueTest
[repro] Exiting with code 0

[...truncated 73 lines...]