You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@lucene.apache.org by Apache Jenkins Server <je...@builds.apache.org> on 2020/02/12 01:56:21 UTC

[JENKINS] Lucene-Solr-Tests-master - Build # 4225 - Unstable

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/4225/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestCollectionAdminRequest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestCollectionAdminRequest:     1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestCollectionAdminRequest: 
   1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([C1FEF1EC5DE91B2A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestCollectionAdminRequest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([C1FEF1EC5DE91B2A]:0)




Build Log:
[...truncated 16082 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.request.TestCollectionAdminRequest
   [junit4]   2> Feb 12, 2020 1:41:35 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 1:41:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestCollectionAdminRequest: 
   [junit4]   2>    1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 1:41:55 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 1:41:58 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5bf28670), locale=bez, timezone=America/Montreal
   [junit4]   2> NOTE: Linux 4.15.0-54-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=4,threads=2,free=213594112,total=264241152
   [junit4]   2> NOTE: All tests run in this JVM: [TestCollectionAdminRequest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestCollectionAdminRequest -Dtests.seed=C1FEF1EC5DE91B2A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=bez -Dtests.timezone=America/Montreal -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | TestCollectionAdminRequest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestCollectionAdminRequest: 
   [junit4]    >    1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C1FEF1EC5DE91B2A]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=15, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestCollectionAdminRequest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([C1FEF1EC5DE91B2A]:0)
   [junit4] Completed [12/207 (1!)] on J2 in 23.46s, 5 tests, 2 errors <<< FAILURES!

[...truncated 51921 lines...]

[JENKINS] Lucene-Solr-Tests-master - Build # 4229 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/4229/

10 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.util.CircularListTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.util.CircularListTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.util.CircularListTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.util.CircularListTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.util.DOMUtilTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.util.DOMUtilTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.util.DOMUtilTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.util.DOMUtilTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-EmpiricalDistributionEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-EmpiricalDistributionEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=RUNNABLE, group=TGRP-EmpiricalDistributionEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=RUNNABLE, group=TGRP-EmpiricalDistributionEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)




Build Log:
[...truncated 12495 lines...]
   [junit4] Suite: org.apache.solr.util.CircularListTest
   [junit4]   2> Feb 12, 2020 1:40:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 1:40:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.util.CircularListTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 1:40:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 1:41:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=171, maxMBSortInHeap=5.508747575222465, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@54b46f4d), locale=bo-CN, timezone=CAT
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=488695096,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [CircularListTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CircularListTest -Dtests.seed=7E84E10FB0680F8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=bo-CN -Dtests.timezone=CAT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | CircularListTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.util.CircularListTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CircularListTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)
   [junit4] Completed [4/904 (1!)] on J1 in 23.60s, 1 test, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.util.DOMUtilTest
   [junit4]   2> Feb 12, 2020 1:40:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 1:40:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.util.DOMUtilTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 1:40:57 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 1:41:00 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=623, maxMBSortInHeap=7.586260944348938, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@6fa2299f), locale=ebu-KE, timezone=SystemV/CST6CDT
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=471440992,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [DOMUtilTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DOMUtilTest -Dtests.seed=7E84E10FB0680F8 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ebu-KE -Dtests.timezone=SystemV/CST6CDT -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | DOMUtilTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.util.DOMUtilTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DOMUtilTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7E84E10FB0680F8]:0)
   [junit4] Completed [5/904 (2!)] on J2 in 23.56s, 1 test, 2 errors <<< FAILURES!

[...truncated 3420 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest
   [junit4]   2> Feb 12, 2020 2:10:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 2:10:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 2:10:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 2:10:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@2efed490), locale=ebu-KE, timezone=NZ
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=476962984,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [DivideEvaluatorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=DivideEvaluatorTest -Dtests.seed=BC30C34788D85878 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ebu-KE -Dtests.timezone=NZ -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J1 | DivideEvaluatorTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.DivideEvaluatorTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-DivideEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)
   [junit4] Completed [1/207 (1!)] on J1 in 23.73s, 11 tests, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest
   [junit4]   2> Feb 12, 2020 2:10:15 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 2:10:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 2:10:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 2:10:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@5a126699), locale=tg, timezone=Canada/Newfoundland
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=474808344,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [CumulativeProbabilityEvaluatorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CumulativeProbabilityEvaluatorTest -Dtests.seed=BC30C34788D85878 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=tg -Dtests.timezone=Canada/Newfoundland -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J0 | CumulativeProbabilityEvaluatorTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.CumulativeProbabilityEvaluatorTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-CumulativeProbabilityEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)
   [junit4] Completed [2/207 (2!)] on J0 in 23.76s, 1 test, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest
   [junit4]   2> Feb 12, 2020 2:10:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 2:10:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-EmpiricalDistributionEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 2:10:35 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-EmpiricalDistributionEvaluatorTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 2:10:37 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=RUNNABLE, group=TGRP-EmpiricalDistributionEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFieldsData(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=10, maxDocsPerChunk=252, blockShift=7), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=10, blockSize=7)), sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@65ea1fdc), locale=ta-SG, timezone=Antarctica/Macquarie
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=489380616,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [EmpiricalDistributionEvaluatorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=EmpiricalDistributionEvaluatorTest -Dtests.seed=BC30C34788D85878 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=ta-SG -Dtests.timezone=Antarctica/Macquarie -Dtests.asserts=true -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR   0.00s J2 | EmpiricalDistributionEvaluatorTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.EmpiricalDistributionEvaluatorTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-EmpiricalDistributionEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=RUNNABLE, group=TGRP-EmpiricalDistributionEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([BC30C34788D85878]:0)
   [junit4] Completed [3/207 (3!)] on J2 in 23.81s, 1 test, 2 errors <<< FAILURES!

[...truncated 51942 lines...]

[JENKINS] Lucene-Solr-Tests-master - Build # 4228 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/4228/

6 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)




Build Log:
[...truncated 16086 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest
   [junit4]   2> Feb 12, 2020 12:56:51 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 12:57:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 12:57:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 12:57:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=202, maxMBSortInHeap=7.939958292165882, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@62ba482), locale=kok, timezone=America/Tijuana
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=475362288,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [GreaterThanEqualToEvaluatorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=GreaterThanEqualToEvaluatorTest -Dtests.seed=7CFB1ADF65960A0A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=kok -Dtests.timezone=America/Tijuana -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | GreaterThanEqualToEvaluatorTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEqualToEvaluatorTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEqualToEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)
   [junit4] Completed [1/207 (1!)] on J1 in 23.60s, 7 tests, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest
   [junit4]   2> Feb 12, 2020 12:56:51 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 12:57:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 12:57:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 12:57:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Lucene84, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@12aaf31e), locale=mfe-MU, timezone=America/Merida
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=476516448,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [GreaterThanEvaluatorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=GreaterThanEvaluatorTest -Dtests.seed=7CFB1ADF65960A0A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=mfe-MU -Dtests.timezone=America/Merida -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J2 | GreaterThanEvaluatorTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.GreaterThanEvaluatorTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-GreaterThanEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)
   [junit4] Completed [2/207 (2!)] on J2 in 23.56s, 7 tests, 2 errors <<< FAILURES!

[...truncated 1 lines...]
   [junit4] Suite: org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest
   [junit4]   2> Feb 12, 2020 12:56:51 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> WARNING: Will linger awaiting termination of 1 leaked thread(s).
   [junit4]   2> Feb 12, 2020 12:57:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl checkThreadLeaks
   [junit4]   2> SEVERE: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest: 
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 12:57:11 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 12:57:14 PM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=CheapBastard, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@c1287c1), locale=fil-PH, timezone=Australia/Sydney
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=476339944,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [FloorEvaluatorTest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=FloorEvaluatorTest -Dtests.seed=7CFB1ADF65960A0A -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=fil-PH -Dtests.timezone=Australia/Sydney -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J0 | FloorEvaluatorTest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.io.stream.eval.FloorEvaluatorTest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-FloorEvaluatorTest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([7CFB1ADF65960A0A]:0)
   [junit4] Completed [3/207 (3!)] on J0 in 23.58s, 5 tests, 2 errors <<< FAILURES!

[...truncated 51942 lines...]

[JENKINS] Lucene-Solr-Tests-master - Build # 4227 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/4227/

4 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi

Error Message:
expected:<2> but was:<3>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<3>
	at __randomizedtesting.SeedInfo.seed([7D9E2931BD191A5A:2AB7D28466EBF841]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:631)
	at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testReadApi(AutoScalingHandlerTest.java:898)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testSuggestionsWithPayload

Error Message:


Stack Trace:
java.lang.AssertionError
	at __randomizedtesting.SeedInfo.seed([7D9E2931BD191A5A:2452243CE151B305]:0)
	at org.junit.Assert.fail(Assert.java:86)
	at org.junit.Assert.assertTrue(Assert.java:41)
	at org.junit.Assert.assertTrue(Assert.java:52)
	at org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest.testSuggestionsWithPayload(AutoScalingHandlerTest.java:129)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestDelegationTokenRequest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestDelegationTokenRequest:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestDelegationTokenRequest: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([1FF77B2D50A7AA77]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestDelegationTokenRequest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([1FF77B2D50A7AA77]:0)




Build Log:
[...truncated 14420 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.AutoScalingHandlerTest
   [junit4]   2> 3120033 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 3120033 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.SolrTestCaseJ4 Created dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/data-dir-192-001
   [junit4]   2> 3120034 WARN  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=14 numCloses=14
   [junit4]   2> 3120034 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 3120035 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 3120036 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.MiniSolrCloudCluster Starting cluster of 2 servers in /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001
   [junit4]   2> 3120036 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 3120036 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 3120037 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 3120137 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.ZkTestServer start zk server on port:39992
   [junit4]   2> 3120137 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:39992
   [junit4]   2> 3120137 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:39992
   [junit4]   2> 3120137 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 39992
   [junit4]   2> 3120139 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3120141 INFO  (zkConnectionManagerCallback-13083-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3120141 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3120144 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3120145 INFO  (zkConnectionManagerCallback-13085-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3120145 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3120145 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3120147 INFO  (zkConnectionManagerCallback-13087-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3120147 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3120254 WARN  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 3120255 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 3120255 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 3120255 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 3120255 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3120255 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3120255 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3120256 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@662ec86f{/solr,null,AVAILABLE}
   [junit4]   2> 3120256 WARN  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 3120256 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 3120256 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 3120256 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@523ecf50{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:37567}
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.e.j.s.Server Started @3120295ms
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=37567}
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 3120257 ERROR (jetty-launcher-13088-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@60a4dd57{/solr,null,AVAILABLE}
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 3120257 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T10:16:33.387761Z
   [junit4]   2> 3120258 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.AbstractConnector Started ServerConnector@28cd8d6d{ssl,[ssl, alpn, http/1.1, h2]}{127.0.0.1:36920}
   [junit4]   2> 3120258 INFO  (jetty-launcher-13088-thread-2) [     ] o.e.j.s.Server Started @3120297ms
   [junit4]   2> 3120258 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/solr, hostPort=36920}
   [junit4]   2> 3120259 ERROR (jetty-launcher-13088-thread-2) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 3120259 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 3120259 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 3120259 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 3120259 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 3120259 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T10:16:33.389171Z
   [junit4]   2> 3120270 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3120271 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3120271 INFO  (zkConnectionManagerCallback-13090-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3120271 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3120272 INFO  (zkConnectionManagerCallback-13092-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3120272 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3120272 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3120273 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.s.SolrDispatchFilter solr.xml found in ZooKeeper. Loading...
   [junit4]   2> 3121413 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 3121427 WARN  (jetty-launcher-13088-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@3f00598c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3121427 WARN  (jetty-launcher-13088-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@3f00598c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3121430 WARN  (jetty-launcher-13088-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@3dacaa9c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3121430 WARN  (jetty-launcher-13088-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@3dacaa9c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3121432 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39992/solr
   [junit4]   2> 3121432 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3121434 INFO  (zkConnectionManagerCallback-13100-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3121442 INFO  (jetty-launcher-13088-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3121555 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3121577 INFO  (zkConnectionManagerCallback-13102-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3121577 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3121864 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:36920_solr
   [junit4]   2> 3121865 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.Overseer Overseer (id=72272429483884550-127.0.0.1:36920_solr-n_0000000000) starting
   [junit4]   2> 3121871 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:36920_solr
   [junit4]   2> 3121888 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36920_solr
   [junit4]   2> 3121890 INFO  (zkCallback-13101-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3121892 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 3121892 WARN  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 3121960 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3122013 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36920.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3122034 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36920.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3122034 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36920.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3122036 INFO  (jetty-launcher-13088-thread-2) [n:127.0.0.1:36920_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node2/.
   [junit4]   2> 3124470 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true]
   [junit4]   2> 3124472 WARN  (jetty-launcher-13088-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@1f6b08d8[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3124472 WARN  (jetty-launcher-13088-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@1f6b08d8[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3124492 WARN  (jetty-launcher-13088-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@798f9c93[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3124492 WARN  (jetty-launcher-13088-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@798f9c93[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 3124510 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:39992/solr
   [junit4]   2> 3124511 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3124512 INFO  (zkConnectionManagerCallback-13112-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3124512 INFO  (jetty-launcher-13088-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3124615 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3124616 INFO  (zkConnectionManagerCallback-13114-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3124616 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3124637 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 3124642 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.ZkController Publish node=127.0.0.1:37567_solr as DOWN
   [junit4]   2> 3124659 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3124659 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37567_solr
   [junit4]   2> 3124660 INFO  (zkCallback-13101-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3124660 INFO  (zkCallback-13113-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 3124662 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 3124662 WARN  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 3124729 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 3124832 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37567.solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3124875 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37567.solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3124875 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37567.solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3124877 INFO  (jetty-launcher-13088-thread-1) [n:127.0.0.1:37567_solr     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node1/.
   [junit4]   2> 3124987 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.MiniSolrCloudCluster waitForAllNodes: numServers=2
   [junit4]   2> 3125000 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 3125001 INFO  (zkConnectionManagerCallback-13123-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 3125001 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 3125003 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 3125020 INFO  (SUITE-AutoScalingHandlerTest-seed#[7D9E2931BD191A5A]-worker) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:39992/solr ready
   [junit4]   2> 3125090 INFO  (TEST-AutoScalingHandlerTest.testErrorHandling-seed#[7D9E2931BD191A5A]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testErrorHandling
   [junit4]   2> 3125316 ERROR (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-cluster-policy=[{cores=<10, node=#ANY}, {shard=#EACH, node=#ANY}, {nodeRole=overseer, replica=0}], errorMessages=[replica is required in {
   [junit4]   2>   "shard":"#EACH",
   [junit4]   2>   "node":"#ANY"}]}], 
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:241)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:178)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:146)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 3125316 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=400 QTime=18
   [junit4]   2> 3125319 INFO  (TEST-AutoScalingHandlerTest.testErrorHandling-seed#[7D9E2931BD191A5A]) [     ] o.a.s.SolrTestCaseJ4 ###Ending testErrorHandling
   [junit4]   2> 3125375 INFO  (TEST-AutoScalingHandlerTest.testValidation-seed#[7D9E2931BD191A5A]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testValidation
   [junit4]   2> 3125573 ERROR (qtp499398819-39895) [n:127.0.0.1:37567_solr     ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-trigger={name=node_lost_trigger, event=nodeLost, waitFor=10m, enabled=true, foo=bar, actions=[{name=compute_plan, class=solr.ComputePlanAction}]}, errorMessages=[Error validating trigger config node_lost_trigger: TriggerValidationException{name=node_lost_trigger, details='{foo=unknown property}'}]}], 
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:241)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:178)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209)
   [junit4]   2> 	at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> 	at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:343)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$1.run(SslConnection.java:146)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 3125573 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=400 QTime=8
   [junit4]   2> 3125600 ERROR (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-trigger={name=search_rate_trigger, event=searchRate, waitFor=10m, enabled=true, aboveRate=foo, actions=[{name=compute_plan, class=solr.ComputePlanAction}]}, errorMessages=[Error validating trigger config search_rate_trigger: TriggerValidationException{name=search_rate_trigger, details='{aboveRate=Invalid configuration value: 'foo': java.lang.NumberFormatException: For input string: "foo"}'}]}], 
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:241)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:178)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 3125600 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=400 QTime=25
   [junit4]   2> 3125603 ERROR (qtp499398819-39897) [n:127.0.0.1:37567_solr     ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-trigger={name=node_lost_trigger, event=nodeLost, waitFor=10m, enabled=true, actions=[{name=compute_plan, foo=bar, class=solr.ComputePlanAction}]}, errorMessages=[Error validating trigger config node_lost_trigger: TriggerValidationException{name=compute_plan, details='{foo=unknown property}'}]}], 
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:241)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:178)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209)
   [junit4]   2> 	at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> 	at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:343)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 3125603 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=400 QTime=1
   [junit4]   2> 3125665 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3125698 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125699 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=82
   [junit4]   2> 3125703 WARN  (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.c.a.AutoScalingHandler invalid listener configuration
   [junit4]   2>           => TriggerValidationException{name=xyz, details='{foo=unknown property}'}
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.TriggerListenerBase.configure(TriggerListenerBase.java:74)
   [junit4]   2> org.apache.solr.cloud.autoscaling.TriggerValidationException: null
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.TriggerListenerBase.configure(TriggerListenerBase.java:74) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.HttpTriggerListener.configure(HttpTriggerListener.java:78) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleSetListener(AutoScalingHandler.java:498) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:209) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:178) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:343) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419) ~[java/:?]
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351) ~[java/:?]
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166) ~[java/:?]
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604) ~[jetty-servlet-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545) ~[jetty-servlet-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485) ~[jetty-servlet-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322) ~[jetty-rewrite-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:500) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270) ~[jetty-server-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117) ~[jetty-io-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806) ~[jetty-util-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938) ~[jetty-util-9.4.24.v20191120.jar:9.4.24.v20191120]
   [junit4]   2> 	at java.lang.Thread.run(Thread.java:834) [?:?]
   [junit4]   2> 3125704 ERROR (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.h.RequestHandlerBase org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, errors: [{set-listener={name=xyz, trigger=node_lost_trigger, stage=[STARTED, ABORTED, SUCCEEDED], foo=bar, beforeAction=execute_plan, class=org.apache.solr.cloud.autoscaling.HttpTriggerListener, url=http://xyz.com/on_node_lost?node={$LOST_NODE_NAME}}, errorMessages=[invalid listener configuration: TriggerValidationException{name=xyz, details='{foo=unknown property}'}]}], 
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:241)
   [junit4]   2> 	at org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:178)
   [junit4]   2> 	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:209)
   [junit4]   2> 	at org.apache.solr.api.ApiBag$ReqHandlerToApi.call(ApiBag.java:242)
   [junit4]   2> 	at org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:343)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
   [junit4]   2> 	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
   [junit4]   2> 	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:166)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
   [junit4]   2> 	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
   [junit4]   2> 	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:717)
   [junit4]   2> 	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
   [junit4]   2> 	at org.eclipse.jetty.server.Server.handle(Server.java:500)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
   [junit4]   2> 	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
   [junit4]   2> 	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:543)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:398)
   [junit4]   2> 	at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:161)
   [junit4]   2> 	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
   [junit4]   2> 	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
   [junit4]   2> 	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> 
   [junit4]   2> 3125704 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=400 QTime=3
   [junit4]   2> 3125705 INFO  (TEST-AutoScalingHandlerTest.testValidation-seed#[7D9E2931BD191A5A]) [     ] o.a.s.SolrTestCaseJ4 ###Ending testValidation
   [junit4]   2> 3125801 INFO  (TEST-AutoScalingHandlerTest.testReadApi-seed#[7D9E2931BD191A5A]) [     ] o.a.s.SolrTestCaseJ4 ###Starting testReadApi
   [junit4]   2> 3125805 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125820 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125820 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=18
   [junit4]   2> 3125835 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125837 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName&wt=javabin&version=2&group=solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125851 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=29
   [junit4]   2> 3125869 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3125871 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125888 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/autoscaling params={wt=javabin&version=2} status=0 QTime=35
   [junit4]   2> 3125908 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3125910 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3125929 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=40
   [junit4]   2> 3125930 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=0
   [junit4]   2> 3125935 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3125953 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3125954 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling/diagnostics params={wt=javabin&version=2} status=0 QTime=22
   [junit4]   2> 3126001 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3126037 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=0
   [junit4]   2> 3126039 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/cluster/autoscaling params={wt=javabin&version=2} status=0 QTime=82
   [junit4]   2> 3126040 INFO  (qtp1731687872-39904) [n:127.0.0.1:36920_solr     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf&maxShardsPerNode=3&name=readApiTestViolations&nrtReplicas=6&action=CREATE&numShards=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 3126060 INFO  (OverseerThreadFactory-9945-thread-1-processing-n:127.0.0.1:36920_solr) [n:127.0.0.1:36920_solr     ] o.a.s.c.a.c.CreateCollectionCmd Create collection readApiTestViolations
   [junit4]   2> 3126180 WARN  (OverseerThreadFactory-9945-thread-1-processing-n:127.0.0.1:36920_solr) [n:127.0.0.1:36920_solr     ] o.a.s.c.a.c.CreateCollectionCmd Specified number of replicas of 6 on collection readApiTestViolations is higher than the number of Solr instances currently live or live and part of your createNodeSet(2). It's unusual to run two replica of the same slice on the same Solr-instance.
   [junit4]   2> 3126184 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3126186 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/metrics params={prefix=CONTAINER.fs.usableSpace,CONTAINER.fs.totalSpace,CORE.coreName,os.systemLoadAverage,memory.heap.usage&wt=javabin&version=2&group=solr.jvm,solr.node,solr.core} status=0 QTime=1
   [junit4]   2> 3126202 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"readApiTestViolations",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"readApiTestViolations_shard1_replica_n1",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36920/solr",
   [junit4]   2>   "node_name":"127.0.0.1:36920_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 3126229 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"readApiTestViolations",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"readApiTestViolations_shard1_replica_n2",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37567/solr",
   [junit4]   2>   "node_name":"127.0.0.1:37567_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 3126232 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"readApiTestViolations",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"readApiTestViolations_shard1_replica_n4",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36920/solr",
   [junit4]   2>   "node_name":"127.0.0.1:36920_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 3126235 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"readApiTestViolations",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"readApiTestViolations_shard1_replica_n6",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37567/solr",
   [junit4]   2>   "node_name":"127.0.0.1:37567_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 3126246 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"readApiTestViolations",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"readApiTestViolations_shard1_replica_n8",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:36920/solr",
   [junit4]   2>   "node_name":"127.0.0.1:36920_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 3126257 INFO  (OverseerStateUpdate-72272429483884550-127.0.0.1:36920_solr-n_0000000000) [n:127.0.0.1:36920_solr     ] o.a.s.c.o.SliceMutator createReplica() {
   [junit4]   2>   "operation":"ADDREPLICA",
   [junit4]   2>   "collection":"readApiTestViolations",
   [junit4]   2>   "shard":"shard1",
   [junit4]   2>   "core":"readApiTestViolations_shard1_replica_n10",
   [junit4]   2>   "state":"down",
   [junit4]   2>   "base_url":"https://127.0.0.1:37567/solr",
   [junit4]   2>   "node_name":"127.0.0.1:37567_solr",
   [junit4]   2>   "type":"NRT",
   [junit4]   2>   "waitForFinalState":"false"} 
   [junit4]   2> 3126460 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr    x:readApiTestViolations_shard1_replica_n1 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node3&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n1&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3126460 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr    x:readApiTestViolations_shard1_replica_n2 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node5&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n2&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3126570 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr    x:readApiTestViolations_shard1_replica_n1 ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 transient cores
   [junit4]   2> 3126603 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr    x:readApiTestViolations_shard1_replica_n4 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node7&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n4&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3126626 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr    x:readApiTestViolations_shard1_replica_n6 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node9&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n6&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3126821 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 3126824 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n1 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 3126866 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node9 x:readApiTestViolations_shard1_replica_n6 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 3126868 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node7 x:readApiTestViolations_shard1_replica_n4 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 3126903 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2 ] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n2] Schema name=minimal
   [junit4]   2> 3126926 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2 ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3126927 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2 ] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n2' using configuration from configset conf, trusted=true
   [junit4]   2> 3126927 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37567.solr.core.readApiTestViolations.shard1.replica_n2' (registry 'solr.core.readApiTestViolations.shard1.replica_n2') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3126927 INFO  (qtp499398819-39896) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node5 x:readApiTestViolations_shard1_replica_n2 ] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n2] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node1/readApiTestViolations_shard1_replica_n2], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node1/./readApiTestViolations_shard1_replica_n2/data/]
   [junit4]   2> 3126931 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node9 x:readApiTestViolations_shard1_replica_n6 ] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n6] Schema name=minimal
   [junit4]   2> 3126938 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n1 ] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n1] Schema name=minimal
   [junit4]   2> 3126953 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node9 x:readApiTestViolations_shard1_replica_n6 ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3126953 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node9 x:readApiTestViolations_shard1_replica_n6 ] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n6' using configuration from configset conf, trusted=true
   [junit4]   2> 3126954 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node9 x:readApiTestViolations_shard1_replica_n6 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_37567.solr.core.readApiTestViolations.shard1.replica_n6' (registry 'solr.core.readApiTestViolations.shard1.replica_n6') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3126954 INFO  (qtp499398819-39895) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node9 x:readApiTestViolations_shard1_replica_n6 ] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n6] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node1/readApiTestViolations_shard1_replica_n6], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node1/./readApiTestViolations_shard1_replica_n6/data/]
   [junit4]   2> 3126957 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n1 ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3126957 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n1 ] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n1' using configuration from configset conf, trusted=true
   [junit4]   2> 3126958 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36920.solr.core.readApiTestViolations.shard1.replica_n1' (registry 'solr.core.readApiTestViolations.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3126958 INFO  (qtp1731687872-39905) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node3 x:readApiTestViolations_shard1_replica_n1 ] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node2/readApiTestViolations_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node2/./readApiTestViolations_shard1_replica_n1/data/]
   [junit4]   2> 3126987 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr    x:readApiTestViolations_shard1_replica_n10 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node12&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n10&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3126991 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node7 x:readApiTestViolations_shard1_replica_n4 ] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n4] Schema name=minimal
   [junit4]   2> 3126994 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr    x:readApiTestViolations_shard1_replica_n8 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&coreNodeName=core_node11&collection.configName=conf&newCollection=true&name=readApiTestViolations_shard1_replica_n8&action=CREATE&numShards=1&collection=readApiTestViolations&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 3127015 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node7 x:readApiTestViolations_shard1_replica_n4 ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3127015 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node7 x:readApiTestViolations_shard1_replica_n4 ] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n4' using configuration from configset conf, trusted=true
   [junit4]   2> 3127015 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node7 x:readApiTestViolations_shard1_replica_n4 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36920.solr.core.readApiTestViolations.shard1.replica_n4' (registry 'solr.core.readApiTestViolations.shard1.replica_n4') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3127015 INFO  (qtp1731687872-39906) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node7 x:readApiTestViolations_shard1_replica_n4 ] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n4] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node2/readApiTestViolations_shard1_replica_n4], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node2/./readApiTestViolations_shard1_replica_n4/data/]
   [junit4]   2> 3127104 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node11 x:readApiTestViolations_shard1_replica_n8 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 3127136 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node12 x:readApiTestViolations_shard1_replica_n10 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 3127171 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node11 x:readApiTestViolations_shard1_replica_n8 ] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n8] Schema name=minimal
   [junit4]   2> 3127316 INFO  (qtp499398819-39897) [n:127.0.0.1:37567_solr c:readApiTestViolations s:shard1 r:core_node12 x:readApiTestViolations_shard1_replica_n10 ] o.a.s.s.IndexSchema [readApiTestViolations_shard1_replica_n10] Schema name=minimal
   [junit4]   2> 3127317 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node11 x:readApiTestViolations_shard1_replica_n8 ] o.a.s.s.IndexSchema Loaded schema minimal/1.1 with uniqueid field id
   [junit4]   2> 3127317 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node11 x:readApiTestViolations_shard1_replica_n8 ] o.a.s.c.CoreContainer Creating SolrCore 'readApiTestViolations_shard1_replica_n8' using configuration from configset conf, trusted=true
   [junit4]   2> 3127317 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node11 x:readApiTestViolations_shard1_replica_n8 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr_36920.solr.core.readApiTestViolations.shard1.replica_n8' (registry 'solr.core.readApiTestViolations.shard1.replica_n8') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@4d3505fd
   [junit4]   2> 3127317 INFO  (qtp1731687872-39966) [n:127.0.0.1:36920_solr c:readApiTestViolations s:shard1 r:core_node11 x:readApiTestViolations_shard1_replica_n8 ] o.a.s.c.SolrCore [[readApiTestViolations_shard1_replica_n8] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.AutoScalingHandlerTest_7D9E2931BD191A5A-001/tempDir-001/node2/readApiTestViolations_shard1_replica_n8], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-

[...truncated too long message...]

cessEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 10:41:29 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 10:41:32 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Lucene84, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@4e9e423f), locale=uk-UA, timezone=Asia/Calcutta
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=479741040,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [TestDelegationTokenRequest]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDelegationTokenRequest -Dtests.seed=1FF77B2D50A7AA77 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=uk-UA -Dtests.timezone=Asia/Calcutta -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
   [junit4] ERROR   0.00s J2 | TestDelegationTokenRequest (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.request.TestDelegationTokenRequest: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1FF77B2D50A7AA77]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenRequest]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([1FF77B2D50A7AA77]:0)
   [junit4] Completed [10/207 (1!)] on J2 in 23.73s, 3 tests, 2 errors <<< FAILURES!

[...truncated 51923 lines...]

[JENKINS] Lucene-Solr-Tests-master - Build # 4226 - Still Unstable

Posted by Apache Jenkins Server <je...@builds.apache.org>.
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/4226/

3 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test

Error Message:
Timed out waiting for replica core_node12 (1581489845860) to replicate from leader core_node3 (0)

Stack Trace:
java.lang.AssertionError: Timed out waiting for replica core_node12 (1581489845860) to replicate from leader core_node3 (0)
	at __randomizedtesting.SeedInfo.seed([AFECDFD3FAF6A71C:27B8E009540ACAE4]:0)
	at org.junit.Assert.fail(Assert.java:88)
	at org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForReplicationFromReplicas(AbstractFullDistribZkTestBase.java:2294)
	at org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest.test(ChaosMonkeySafeLeaderWithPullReplicasTest.java:211)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
	at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
	at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
	at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
	at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
	at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
	at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
	at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
	at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
	at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
	at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.response.TestDelegationTokenResponse

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.client.solrj.response.TestDelegationTokenResponse:     1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.response.TestDelegationTokenResponse: 
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([AA11AF9073FF42A0]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.client.solrj.response.TestDelegationTokenResponse

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]
        at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
        at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
        at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
        at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
        at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
        at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
        at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
	at __randomizedtesting.SeedInfo.seed([AA11AF9073FF42A0]:0)




Build Log:
[...truncated 12624 lines...]
   [junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest
   [junit4]   2> 98999 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[AFECDFD3FAF6A71C]-worker) [     ] o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 99008 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[AFECDFD3FAF6A71C]-worker) [     ] o.a.s.SolrTestCaseJ4 Created dataDir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/data-dir-19-001
   [junit4]   2> 99008 WARN  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[AFECDFD3FAF6A71C]-worker) [     ] o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=1 numCloses=1
   [junit4]   2> 99008 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[AFECDFD3FAF6A71C]-worker) [     ] o.a.s.SolrTestCaseJ4 Using TrieFields (NUMERIC_POINTS_SYSPROP=false) w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 99009 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[AFECDFD3FAF6A71C]-worker) [     ] o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: @org.apache.solr.util.RandomizeSSL(reason="", value=0.0/0.0, ssl=0.0/0.0, clientAuth=0.0/0.0)
   [junit4]   2> 99009 INFO  (SUITE-ChaosMonkeySafeLeaderWithPullReplicasTest-seed#[AFECDFD3FAF6A71C]-worker) [     ] o.a.s.BaseDistributedSearchTestCase Setting hostContext system property: /r_zd/zi
   [junit4]   2> 99010 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ChaosMonkeySafeLeaderWithPullReplicasTest Starting ChaosMonkey test with 2 shards and 6 nodes
   [junit4]   2> 99022 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 99023 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 99023 INFO  (ZkTestServer Run Thread) [     ] o.a.s.c.ZkTestServer Starting server
   [junit4]   2> 99123 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer start zk server on port:45891
   [junit4]   2> 99123 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer waitForServerUp: 127.0.0.1:45891
   [junit4]   2> 99123 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer parse host and port list: 127.0.0.1:45891
   [junit4]   2> 99123 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer connecting to 127.0.0.1 45891
   [junit4]   2> 99125 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 99127 INFO  (zkConnectionManagerCallback-390-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 99127 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 99130 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 99132 INFO  (zkConnectionManagerCallback-392-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 99132 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 99135 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 99137 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/schema15.xml to /configs/conf1/schema.xml
   [junit4]   2> 99139 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/solrconfig.snippet.randomindexconfig.xml to /configs/conf1/solrconfig.snippet.randomindexconfig.xml
   [junit4]   2> 99140 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 99142 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 99143 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 99144 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/enumsConfig.xml to /configs/conf1/enumsConfig.xml
   [junit4]   2> 99154 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 99155 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 99156 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 99170 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkTestServer put /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/core/src/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 99171 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase Will use NRT replicas unless explicitly asked otherwise
   [junit4]   2> 99552 WARN  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 99552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 99552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 99552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 99553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 99553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 99553 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 99563 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@1931bb47{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 99563 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.AbstractConnector Started ServerConnector@7d4bb22b{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:38721}
   [junit4]   2> 99563 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.s.Server Started @99601ms
   [junit4]   2> 99563 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/control/data, hostPort=38721, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/control-001/cores, replicaType=NRT}
   [junit4]   2> 99564 ERROR (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 99564 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 99564 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 99564 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 99564 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 99564 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T06:42:37.387476Z
   [junit4]   2> 99565 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 99579 INFO  (zkConnectionManagerCallback-394-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 99579 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 99681 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 99682 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/control-001/solr.xml
   [junit4]   2> 99702 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 99702 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 99703 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 102737 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 102738 WARN  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@6a7b1b4f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 102738 WARN  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@6a7b1b4f[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 102754 WARN  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@29eca5c2[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 102755 WARN  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@29eca5c2[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 102756 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45891/solr
   [junit4]   2> 102758 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 102759 INFO  (zkConnectionManagerCallback-401-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 102759 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 102863 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 102864 INFO  (zkConnectionManagerCallback-403-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 102864 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 103023 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.OverseerElectionContext I am going to be the leader 127.0.0.1:38721_r_zd%2Fzi
   [junit4]   2> 103024 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.Overseer Overseer (id=72271588242751492-127.0.0.1:38721_r_zd%2Fzi-n_0000000000) starting
   [junit4]   2> 103030 INFO  (OverseerStateUpdate-72271588242751492-127.0.0.1:38721_r_zd%2Fzi-n_0000000000) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.Overseer Starting to work on the main queue : 127.0.0.1:38721_r_zd%2Fzi
   [junit4]   2> 103031 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38721_r_zd%2Fzi
   [junit4]   2> 103033 INFO  (zkCallback-402-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 103035 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 103036 WARN  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 103107 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 103164 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 103188 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 103188 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 103202 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/control-001/cores
   [junit4]   2> 103250 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 103250 INFO  (zkConnectionManagerCallback-412-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 103251 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 103252 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 103254 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45891/solr ready
   [junit4]   2> 103265 INFO  (qtp593696977-1155) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38721_r_zd%252Fzi&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 103277 INFO  (OverseerThreadFactory-474-thread-1-processing-n:127.0.0.1:38721_r_zd%2Fzi) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.a.c.CreateCollectionCmd Create collection control_collection
   [junit4]   2> 103386 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi    x:control_collection_shard1_replica_n1 ] o.a.s.h.a.CoreAdminOperation core create command qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT
   [junit4]   2> 103387 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi    x:control_collection_shard1_replica_n1 ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 104403 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrConfig Using Lucene MatchVersion: 9.0.0
   [junit4]   2> 104476 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.IndexSchema [control_collection_shard1_replica_n1] Schema name=test
   [junit4]   2> 104479 WARN  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieIntField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 104501 WARN  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieFloatField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 104546 WARN  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class [solr.TrieDateField]. Please consult documentation how to replace it accordingly.
   [junit4]   2> 104750 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.IndexSchema Loaded schema test/1.6 with uniqueid field id
   [junit4]   2> 104838 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.CoreContainer Creating SolrCore 'control_collection_shard1_replica_n1' using configuration from configset conf1, trusted=true
   [junit4]   2> 104839 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.core.control_collection.shard1.replica_n1' (registry 'solr.core.control_collection.shard1.replica_n1') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 104859 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrCore [[control_collection_shard1_replica_n1] ] Opening new SolrCore at [/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/control-001/cores/control_collection_shard1_replica_n1], dataDir=[/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/control-001/cores/control_collection_shard1_replica_n1/data/]
   [junit4]   2> 104872 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=26, maxMergeAtOnceExplicit=23, maxMergedSegmentMB=24.7861328125, floorSegmentMB=1.7900390625, forceMergeDeletesPctAllowed=18.765870931367544, segmentsPerTier=10.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.3145086449624004, deletesPctAllowed=26.25278566105579
   [junit4]   2> 104930 WARN  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.RequestHandlers INVALID paramSet a in requestHandler {type = requestHandler,name = /dump,class = DumpRequestHandler,attributes = {initParams=a, name=/dump, class=DumpRequestHandler},args = {defaults={a=A, b=B}}}
   [junit4]   2> 105128 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.UpdateHandler Using UpdateLog implementation: org.apache.solr.update.UpdateLog
   [junit4]   2> 105128 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.UpdateLog Initializing UpdateLog: dataDir= defaultSyncLevel=FLUSH numRecordsToKeep=100 maxNumLogsToKeep=10 numVersionBuckets=65536
   [junit4]   2> 105132 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.CommitTracker Hard AutoCommit: if uncommitted for 15000ms; 
   [junit4]   2> 105132 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.CommitTracker Soft AutoCommit: disabled
   [junit4]   2> 105134 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.RandomMergePolicy RandomMergePolicy wrapping class org.apache.lucene.index.TieredMergePolicy: [TieredMergePolicy: maxMergeAtOnce=18, maxMergeAtOnceExplicit=26, maxMergedSegmentMB=104.404296875, floorSegmentMB=1.5498046875, forceMergeDeletesPctAllowed=4.166322676143686, segmentsPerTier=24.0, maxCFSSegmentSizeMB=8.796093022207999E12, noCFSRatio=0.398998709391443, deletesPctAllowed=20.17957603317024
   [junit4]   2> 105135 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.SolrIndexSearcher Opening [Searcher@ef4dc9a[control_collection_shard1_replica_n1] main]
   [junit4]   2> 105158 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Configured ZooKeeperStorageIO with znodeBase: /configs/conf1
   [junit4]   2> 105158 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.r.ManagedResourceStorage Loaded null at path _rest_managed.json using ZooKeeperStorageIO:path=/configs/conf1
   [junit4]   2> 105159 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.h.ReplicationHandler Commits will be reserved for 10000ms.
   [junit4]   2> 105159 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.u.UpdateLog Could not find max version in index or recent updates, using new clock 1658312209708613632
   [junit4]   2> 105164 INFO  (searcherExecutor-479-thread-1-processing-n:127.0.0.1:38721_r_zd%2Fzi x:control_collection_shard1_replica_n1 c:control_collection s:shard1) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SolrCore [control_collection_shard1_replica_n1] Registered new searcher Searcher@ef4dc9a[control_collection_shard1_replica_n1] main{ExitableDirectoryReader(UninvertingDirectoryReader())}
   [junit4]   2> 105183 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ZkShardTerms Successful update of terms at /collections/control_collection/terms/shard1 to Terms{values={core_node2=0}, version=0}
   [junit4]   2> 105183 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase make sure parent is created /collections/control_collection/leaders/shard1
   [junit4]   2> 105208 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
   [junit4]   2> 105208 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
   [junit4]   2> 105208 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync replicas to http://127.0.0.1:38721/r_zd/zi/control_collection_shard1_replica_n1/
   [junit4]   2> 105208 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SyncStrategy Sync Success - now sync replicas to me
   [junit4]   2> 105208 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.SyncStrategy http://127.0.0.1:38721/r_zd/zi/control_collection_shard1_replica_n1/ has no replicas
   [junit4]   2> 105208 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContextBase Creating leader registration node /collections/control_collection/leaders/shard1/leader after winning as /collections/control_collection/leader_elect/shard1/election/72271588242751492-core_node2-n_0000000000
   [junit4]   2> 105211 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ShardLeaderElectionContext I am the new leader: http://127.0.0.1:38721/r_zd/zi/control_collection_shard1_replica_n1/ shard1
   [junit4]   2> 105313 INFO  (zkCallback-402-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 105313 INFO  (zkCallback-402-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 105315 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.c.ZkController I am the leader, no recovery necessary
   [junit4]   2> 105317 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi c:control_collection s:shard1  x:control_collection_shard1_replica_n1 ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/cores params={qt=/admin/cores&collection.configName=conf1&newCollection=true&name=control_collection_shard1_replica_n1&action=CREATE&numShards=1&collection=control_collection&shard=shard1&wt=javabin&version=2&replicaType=NRT} status=0 QTime=1931
   [junit4]   2> 105320 INFO  (qtp593696977-1155) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 105417 INFO  (zkCallback-402-thread-2) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 105417 INFO  (zkCallback-402-thread-1) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 105418 INFO  (zkCallback-402-thread-3) [     ] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDataChanged path:/collections/control_collection/state.json] for collection [control_collection] has occurred - updating... (live nodes size: [1])
   [junit4]   2> 105418 INFO  (qtp593696977-1155) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=control_collection&nrtReplicas=1&action=CREATE&numShards=1&createNodeSet=127.0.0.1:38721_r_zd%252Fzi&wt=javabin&version=2} status=0 QTime=2153
   [junit4]   2> 105419 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase Waiting to see 1 active replicas in collection: control_collection
   [junit4]   2> 105535 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 105536 INFO  (zkConnectionManagerCallback-418-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 105536 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 105550 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 105552 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.s.i.ZkClientClusterStateProvider Cluster at 127.0.0.1:45891/solr ready
   [junit4]   2> 105555 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.ChaosMonkey monkey: init - expire sessions:false cause connection loss:false
   [junit4]   2> 105556 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.h.a.CollectionsHandler Invoked Collection Action :create with params collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2 and sendToOCPQueue=true
   [junit4]   2> 105560 INFO  (OverseerThreadFactory-474-thread-2-processing-n:127.0.0.1:38721_r_zd%2Fzi) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.a.c.CreateCollectionCmd Create collection collection1
   [junit4]   2> 105560 INFO  (OverseerCollectionConfigSetProcessor-72271588242751492-127.0.0.1:38721_r_zd%2Fzi-n_0000000000) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000000 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 105776 WARN  (OverseerThreadFactory-474-thread-2-processing-n:127.0.0.1:38721_r_zd%2Fzi) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.a.c.CreateCollectionCmd It is unusual to create a collection (collection1) without cores.
   [junit4]   2> 105778 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.h.a.CollectionsHandler Wait for new collection to be active for at most 45 seconds. Check all shard replicas
   [junit4]   2> 105779 INFO  (qtp593696977-1154) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.s.HttpSolrCall [admin] webapp=null path=/admin/collections params={collection.configName=conf1&name=collection1&nrtReplicas=1&action=CREATE&numShards=2&createNodeSet=&stateFormat=1&wt=javabin&version=2} status=0 QTime=223
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 105781 INFO  (watches-415-thread-1) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 105781 INFO  (watches-415-thread-1) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrCloudTestCase active slice count: 2 expected:2
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.SolrCloudTestCase active replica count: 0 expected replica count: 0
   [junit4]   2> 105781 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase Creating jetty instances pullReplicaCount=1 numOtherReplicas=4
   [junit4]   2> 106216 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 1 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-1-001 of type NRT for shard2
   [junit4]   2> 106222 WARN  (closeThreadPool-419-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 106222 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 106222 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 106222 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 106235 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 106235 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 106235 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 106236 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6d7bffff{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 106236 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@17b335a1{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:33529}
   [junit4]   2> 106237 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.Server Started @106274ms
   [junit4]   2> 106237 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/jetty1, hostPort=33529, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-1-001/cores, replicaType=NRT}
   [junit4]   2> 106237 ERROR (closeThreadPool-419-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 106237 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 106238 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 106238 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 106238 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 106238 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T06:42:44.061082Z
   [junit4]   2> 106239 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 106240 INFO  (zkConnectionManagerCallback-421-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 106240 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 106342 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 106342 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-1-001/solr.xml
   [junit4]   2> 106346 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 106346 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 106348 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 107578 INFO  (OverseerCollectionConfigSetProcessor-72271588242751492-127.0.0.1:38721_r_zd%2Fzi-n_0000000000) [n:127.0.0.1:38721_r_zd%2Fzi     ] o.a.s.c.OverseerTaskQueue Response ZK path: /overseer/collection-queue-work/qnr-0000000002 doesn't exist.  Requestor may have disconnected from ZooKeeper
   [junit4]   2> 108121 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 2 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-2-001 of type NRT for shard1
   [junit4]   2> 108122 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 108122 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 108122 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 108122 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 108123 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 108124 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 108124 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 108124 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@6554f4a7{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.AbstractConnector Started ServerConnector@14b2173c{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:45545}
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.Server Started @108162ms
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/jetty2, hostPort=45545, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-2-001/cores, replicaType=NRT}
   [junit4]   2> 108125 ERROR (closeThreadPool-419-thread-2) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 108125 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T06:42:45.948804Z
   [junit4]   2> 108126 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 108128 INFO  (zkConnectionManagerCallback-424-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 108128 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 108255 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 108255 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-2-001/solr.xml
   [junit4]   2> 108282 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 108282 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 108304 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 113160 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 113271 WARN  (closeThreadPool-419-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@24fe1f45[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 113271 WARN  (closeThreadPool-419-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@24fe1f45[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 113414 WARN  (closeThreadPool-419-thread-1) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@39d4cf9c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 113414 WARN  (closeThreadPool-419-thread-1) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@39d4cf9c[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 113415 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45891/solr
   [junit4]   2> 113552 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 113553 INFO  (zkConnectionManagerCallback-431-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 113553 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 113676 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 113698 INFO  (zkConnectionManagerCallback-433-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 113698 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 113713 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1)
   [junit4]   2> 113717 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.ZkController Publish node=127.0.0.1:33529_r_zd%2Fzi as DOWN
   [junit4]   2> 113718 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 113718 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33529_r_zd%2Fzi
   [junit4]   2> 113719 INFO  (zkCallback-402-thread-3) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 113721 INFO  (zkCallback-417-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 113737 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 113738 WARN  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 113769 INFO  (zkCallback-432-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (1) -> (2)
   [junit4]   2> 113850 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 113947 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 114032 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 114032 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 114055 INFO  (closeThreadPool-419-thread-1) [n:127.0.0.1:33529_r_zd%2Fzi     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-1-001/cores
   [junit4]   2> 114449 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.AbstractFullDistribZkTestBase waitForLiveNode: 127.0.0.1:33529_r_zd%2Fzi
   [junit4]   2> 114481 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 114524 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@7b591b63[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 114524 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@7b591b63[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 114544 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@491ac59e[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 114544 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@491ac59e[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 114545 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45891/solr
   [junit4]   2> 114546 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 114564 INFO  (zkConnectionManagerCallback-443-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 114564 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 114620 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 3 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-3-001 of type NRT for shard2
   [junit4]   2> 114621 WARN  (closeThreadPool-419-thread-1) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 114621 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 114621 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 114621 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 114623 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 114623 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 114623 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 114623 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@7ceec833{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.AbstractConnector Started ServerConnector@3cc9f28c{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:39646}
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.e.j.s.Server Started @114661ms
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/jetty3, hostPort=39646, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-3-001/cores, replicaType=NRT}
   [junit4]   2> 114624 ERROR (closeThreadPool-419-thread-1) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 114624 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T06:42:52.447698Z
   [junit4]   2> 114625 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 114627 INFO  (zkConnectionManagerCallback-445-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 114627 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 114718 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 114720 INFO  (zkConnectionManagerCallback-447-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 114720 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 114751 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (2)
   [junit4]   2> 114768 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.ZkController Publish node=127.0.0.1:45545_r_zd%2Fzi as DOWN
   [junit4]   2> 114795 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 114795 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-3-001/solr.xml
   [junit4]   2> 114825 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 114825 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 114826 INFO  (closeThreadPool-419-thread-1) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 114828 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 114828 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45545_r_zd%2Fzi
   [junit4]   2> 114830 INFO  (zkCallback-417-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 114830 INFO  (zkCallback-402-thread-3) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 114830 INFO  (zkCallback-432-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 114844 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 114853 WARN  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 114889 INFO  (zkCallback-446-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (2) -> (3)
   [junit4]   2> 115067 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 115228 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 115321 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 115321 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 115323 INFO  (closeThreadPool-419-thread-2) [n:127.0.0.1:45545_r_zd%2Fzi     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-2-001/cores
   [junit4]   2> 115641 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.AbstractFullDistribZkTestBase waitForLiveNode: 127.0.0.1:45545_r_zd%2Fzi
   [junit4]   2> 116850 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 4 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-4-001 of type NRT for shard1
   [junit4]   2> 116851 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 116851 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 116851 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 116851 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 116852 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 116852 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 116852 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 116853 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@5172deb6{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 116853 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.AbstractConnector Started ServerConnector@3fce04c2{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:38768}
   [junit4]   2> 116853 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.Server Started @116891ms
   [junit4]   2> 116853 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/jetty4, hostPort=38768, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-4-001/cores, replicaType=NRT}
   [junit4]   2> 116854 ERROR (closeThreadPool-419-thread-2) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 116854 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 116854 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 116854 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 116854 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 116854 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T06:42:54.677486Z
   [junit4]   2> 116855 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 116857 INFO  (zkConnectionManagerCallback-454-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 116857 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 116971 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 116971 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-4-001/solr.xml
   [junit4]   2> 116975 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 116975 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 116977 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 121871 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=false]
   [junit4]   2> 122038 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@56848528[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 122038 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@56848528[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 122177 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config Trusting all certificates configured for Client@7a909a76[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 122177 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for Client@7a909a76[provider=null,keyStore=null,trustStore=null]
   [junit4]   2> 122178 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.ZkContainer Zookeeper client=127.0.0.1:45891/solr
   [junit4]   2> 122181 INFO  (zkConnectionManagerCallback-461-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 122192 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 122192 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 122245 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 5 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-5-001 of type PULL for shard2
   [junit4]   2> 122247 WARN  (closeThreadPool-419-thread-3) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 122247 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 122247 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 122247 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 122334 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 122351 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 122351 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 122351 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.session node0 Scavenging every 660000ms
   [junit4]   2> 122352 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@31bcf854{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 122352 INFO  (zkConnectionManagerCallback-463-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 122352 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 122367 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (3)
   [junit4]   2> 122370 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.AbstractConnector Started ServerConnector@f6b1c66{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:37698}
   [junit4]   2> 122370 INFO  (closeThreadPool-419-thread-3) [     ] o.e.j.s.Server Started @122408ms
   [junit4]   2> 122370 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/jetty5, hostPort=37698, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/../../../../../../../../../../home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-5-001/cores, replicaType=PULL}
   [junit4]   2> 122379 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.ZkController Publish node=127.0.0.1:38768_r_zd%2Fzi as DOWN
   [junit4]   2> 122381 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 4 transient cores
   [junit4]   2> 122381 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38768_r_zd%2Fzi
   [junit4]   2> 122382 INFO  (zkCallback-446-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 122382 INFO  (zkCallback-402-thread-3) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 122385 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.p.PackageLoader /packages.json updated to version -1
   [junit4]   2> 122386 WARN  (closeThreadPool-419-thread-2) [     ] o.a.s.c.CoreContainer Not all security plugins configured!  authentication=disabled authorization=disabled.  Solr is only as secure as you make it. Consider configuring authentication/authorization before exposing Solr to users internal or external.  See https://s.apache.org/solrsecurity for more info
   [junit4]   2> 122395 INFO  (zkCallback-417-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 122395 INFO  (zkCallback-432-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 122407 ERROR (closeThreadPool-419-thread-3) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 122407 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   [junit4]   2> 122407 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.s.SolrDispatchFilter  ___      _       Welcome to Apache Solr? version 9.0.0
   [junit4]   2> 122407 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _   Starting in cloud mode on port null
   [junit4]   2> 122407 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_|  Install dir: /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr
   [junit4]   2> 122407 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.s.SolrDispatchFilter |___/\___/_|_|    Start time: 2020-02-12T06:43:00.230417Z
   [junit4]   2> 122427 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 122427 INFO  (zkCallback-462-thread-1) [     ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (3) -> (4)
   [junit4]   2> 122468 INFO  (zkConnectionManagerCallback-468-thread-1) [     ] o.a.s.c.c.ConnectionManager zkClient has connected
   [junit4]   2> 122468 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 122486 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.h.a.MetricsHistoryHandler No .system collection, keeping metrics history in memory.
   [junit4]   2> 122595 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.node' (registry 'solr.node') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 122601 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper)
   [junit4]   2> 122601 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.SolrXmlConfig Loading container configuration from /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-5-001/solr.xml
   [junit4]   2> 122622 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverWorkLoopDelay is ignored
   [junit4]   2> 122622 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.SolrXmlConfig Configuration parameter autoReplicaFailoverBadNodeExpiration is ignored
   [junit4]   2> 122624 INFO  (closeThreadPool-419-thread-3) [     ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f, but no JMX reporters were configured - adding default JMX reporter.
   [junit4]   2> 122673 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jvm' (registry 'solr.jvm') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 122673 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.m.r.SolrJmxReporter JMX monitoring for 'solr.jetty' (registry 'solr.jetty') enabled at server: com.sun.jmx.mbeanserver.JmxMBeanServer@6bb6f
   [junit4]   2> 122695 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.CorePropertiesLocator Found 0 core definitions underneath /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-4-001/cores
   [junit4]   2> 123000 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.AbstractFullDistribZkTestBase waitForLiveNode: 127.0.0.1:38768_r_zd%2Fzi
   [junit4]   2> 126661 INFO  (TEST-ChaosMonkeySafeLeaderWithPullReplicasTest.test-seed#[AFECDFD3FAF6A71C]) [     ] o.a.s.c.AbstractFullDistribZkTestBase create jetty 6 in directory /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-6-001 of type PULL for shard1
   [junit4]   2> 126662 WARN  (closeThreadPool-419-thread-2) [     ] o.e.j.s.h.g.GzipHandler minGzipSize of 0 is inefficient for short content, break even is size 23
   [junit4]   2> 126662 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Start Jetty (configured port=0, binding port=0)
   [junit4]   2> 126662 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Trying to start Jetty on port 0 try number 1 ...
   [junit4]   2> 126662 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.Server jetty-9.4.24.v20191120; built: 2019-11-20T21:37:49.771Z; git: 363d5f2df3a8a28de40604320230664b9c793c16; jvm 11.0.4+10-LTS
   [junit4]   2> 126676 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session DefaultSessionIdManager workerName=node0
   [junit4]   2> 126676 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session No SessionScavenger set, using defaults
   [junit4]   2> 126676 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.session node0 Scavenging every 600000ms
   [junit4]   2> 126676 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.h.ContextHandler Started o.e.j.s.ServletContextHandler@8897843{/r_zd/zi,null,AVAILABLE}
   [junit4]   2> 126677 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.AbstractConnector Started ServerConnector@7393f09d{HTTP/1.1,[http/1.1, h2c]}{127.0.0.1:46012}
   [junit4]   2> 126677 INFO  (closeThreadPool-419-thread-2) [     ] o.e.j.s.Server Started @126714ms
   [junit4]   2> 126677 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.c.s.e.JettySolrRunner Jetty properties: {hostContext=/r_zd/zi, solrconfig=solrconfig.xml, solr.data.dir=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/tempDir-001/jetty6, hostPort=46012, coreRootDirectory=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build/solr-core/test/J1/temp/solr.cloud.ChaosMonkeySafeLeaderWithPullReplicasTest_AFECDFD3FAF6A71C-001/shard-6-001/cores, replicaType=PULL}
   [junit4]   2> 126687 ERROR (closeThreadPool-419-thread-2) [     ] o.a.s.u.StartupLoggingUtils Missing Java Option solr.log.dir. Logging may be missing or incomplete.
   [junit4]   2> 126687 INFO  (closeThreadPool-419-thread-2) [     ] o.a.s.s.SolrDispatchFilter Using logger factory org.apache.logging.slf4j.Log4jLoggerFactory
   

[...truncated too long message...]

rocessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 8:04:23 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> INFO: Starting to interrupt leaked threads:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]
   [junit4]   2> AsyncLogger error handling event seq=0, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=1, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=2, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=3, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> AsyncLogger error handling event seq=4, value='null': java.lang.InterruptedException: null
   [junit4]   2> java.lang.InterruptedException
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2056)
   [junit4]   2> 	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2133)
   [junit4]   2> 	at com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2> 	at com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2> 	at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2> 	at java.base/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> Feb 12, 2020 8:04:25 AM com.carrotsearch.randomizedtesting.ThreadLeakControl tryToInterruptAll
   [junit4]   2> SEVERE: There are still zombie threads that couldn't be terminated:
   [junit4]   2>    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]
   [junit4]   2>         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]   2>         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]   2>         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]   2>         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]   2>         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]   2>         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene84): {}, docValues:{}, maxPointsInLeafNode=656, maxMBSortInHeap=5.6525077123465675, sim=Asserting(org.apache.lucene.search.similarities.AssertingSimilarity@15f8c5e5), locale=kn, timezone=America/Indiana/Vevay
   [junit4]   2> NOTE: Linux 4.4.0-170-generic amd64/Oracle Corporation 11.0.4 (64-bit)/cpus=8,threads=2,free=475513360,total=528482304
   [junit4]   2> NOTE: All tests run in this JVM: [TestDelegationTokenResponse]
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDelegationTokenResponse -Dtests.seed=AA11AF9073FF42A0 -Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=kn -Dtests.timezone=America/Indiana/Vevay -Dtests.asserts=true -Dtests.file.encoding=UTF-8
   [junit4] ERROR   0.00s J1 | TestDelegationTokenResponse (suite) <<<
   [junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.client.solrj.response.TestDelegationTokenResponse: 
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AA11AF9073FF42A0]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   [junit4]    >    1) Thread[id=16, name=Log4j2-TF-1-AsyncLoggerConfig-1, state=TIMED_WAITING, group=TGRP-TestDelegationTokenResponse]
   [junit4]    >         at java.base@11.0.4/jdk.internal.misc.Unsafe.park(Native Method)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:234)
   [junit4]    >         at java.base@11.0.4/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   [junit4]    >         at app//com.lmax.disruptor.TimeoutBlockingWaitStrategy.waitFor(TimeoutBlockingWaitStrategy.java:38)
   [junit4]    >         at app//com.lmax.disruptor.ProcessingSequenceBarrier.waitFor(ProcessingSequenceBarrier.java:56)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:159)
   [junit4]    >         at app//com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
   [junit4]    >         at java.base@11.0.4/java.lang.Thread.run(Thread.java:834)
   [junit4]    > 	at __randomizedtesting.SeedInfo.seed([AA11AF9073FF42A0]:0)
   [junit4] Completed [11/207 (1!)] on J1 in 27.11s, 3 tests, 2 errors <<< FAILURES!

[...truncated 51928 lines...]